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Human  Factors  Engineering:  An  Enabler  for 
Military  Transformation  Through  Effective 
Integration  of  Technology  and  Personnel 

George  Galdorisi  and  Glenn  Osga 

SSC  San  Diego 


The  major  institutions  of  American  National  Security  were  designed  in  a  different 
era  to  meet  different  requirements.  All  of  them  must  be  transformed. 

President  George  W.  Bush 

National  Security  Strategy  of  the  United  States 

September  20,  2002  [1] 


INTRODUCTION 

As  the  United  States’  military  transforms,  warfighters  are  increasingly 
turning  to  technologists  to  solve  operational  challenges  with  technolo¬ 
gies.  A  critical  intersection  between  operational  needs  and  technological 
solutions  is  in  the  multi-dimensional  concept  of  command,  control,  com¬ 
munications,  computers,  intelligence,  surveillance,  and  reconnaissance 
(C4ISR). 

Within  the  overarching  discipline  of  C4ISR,  effective  use  of  human- 
systems  integration  technologies  enables  warfighters  to  make  better 
decisions.  These  technologies  present  exciting  possibilities  for  enhancing 
warfighting  effectiveness.  These  technologies  assist  in  a  number  of  ways 
by  enabling  (1)  more  effective  decisions,  (2)  more  timely  decisions,  and 
(3)  an  optimized  number  of  personnel  to  operate  systems. 

Effective  and  timely  decision-making  has  been  observed  and  quantified  in 
recent  SSC  San  Diego  projects  such  as  the  Multi-Modal  Watchstation 
(MMWS)  and  the  Knowledge  Wall/Knowledge  Web  (K-Web).  Software 
associated  with  decision-aiding  and  visualization  reduces  workload  by 
augmenting  or  replacing  manually  intensive  tasks. 

The  cost  impact  of  technology  to  crew  size  is  often  obscured  by  the  lack 
of  one-to-one  correspondence  between  a  software  technology  unit  cost 
and  a  resulting  shipboard  position  change.  Typically,  the  newer  hardware 
technology  is  both  more  capable  and  cheaper  than  the  old.  Software 
development  and  testing  becomes  the  chief  cost  driver.  Cost  tradeoffs 
between  software  and  personnel  could  be  a  significant  factor  in  determin¬ 
ing  feasibility  and  speed  of  technology  transition  to  acquisition.  For  this 
reason,  it  is  important  to  understand  the  manpower  savings  affected  by 
various  human-systems  technologies  as  well  as  the  concomitant  man¬ 
power  costs  associated  with  continuing  to  use  additional  operators  to 
employ  legacy  systems.  Researchers  at  SSC  San  Diego  have  approximated 
manpower  savings  that  can  be  achieved  with  emerging  technology. 


ABSTRACT 

Transformation  of  the  U.S. 
military  requires  new  ways  of 
defining  both  design  and  mis¬ 
sion  processes  to  improve 
warfighting  performance  and 
reduce  system  costs.  New  tech¬ 
nologies  engendered  through 
the  discipline  of  Human 
Factors  Engineering  at  SSC 
San  Diego  enable  warfighters 
to  make  more  effective  deci¬ 
sions  in  a  timelier  manner 
with  fewer  personnel.  While 
the  tradeoffs  between  new 
technologies  and  numbers  of 
operators  needed  are  complex, 
strong  anecdotal  evidence 
suggests  that  these  manpower 
savings  can  be  significant  and 
have  the  potential  to  acceler¬ 
ate  military  transformation. 
The  Human  Factors  Engineer¬ 
ing  community  centered  at 
SSC  San  Diego  has  documented 
and  quantified  the  improved 
mission  effectiveness  of  fewer 
warfighters  operating  enhanced 
combat  systems.  What  is  less 
well  quantified— due  to  a  num¬ 
ber  of  institutional  factors — is 
the  true  life-cycle  cost  of  mili¬ 
tary  operators.  This  paper 
discusses  design  factors  that 
support  reduced  crew  work¬ 
load  and  factors  that  influence 
crew  cost  estimation  and  size. 
The  conclusion  is  that 
although  researchers  at  SSC 
San  Diego  have  identified 
good  candidate  designs  to  sup¬ 
port  reduced  crew  workload, 
we  cannot  adequately  trade  off 
their  cost  with  personnel  costs 
until  we  can  more  accurately 
quantify  these  personnel  costs. 
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Research  regarding  the  "true  cost"  of  military  personnel  is  less  well  quan¬ 
tified  and  therefore  not  well  understood.  However,  this  understanding  is 
crucial  if  we  are  to  transform  the  U.S.  military  and  effectively  use  tech¬ 
nology  to  enable  manpower  savings. 

TRANSFORMATION:  MAN  AND  MACHINE 

Transforming  the  United  States  Military 

Transformation  of  the  military  has  been  a  strong  theme  of  President 
George  W.  Bush  since  well  before  his  term  began  in  January  2001. 
Candidate  Bush  signaled  the  course  for  transformation  in  a  speech  in 
September  1999  [2]  where  he  stated:  "I  know  that  transforming  our  mili¬ 
tary  is  a  massive  undertaking.... The  real  goal  is  to  move  beyond  marginal 
improvements  — to  replace  existing  programs  with  new  technologies  and 
strategies. ..to  use  this  window  of  opportunity  to  skip  a  generation  of 
weapons  systems."  The  Secretary  of  Defense  2002  Annual  Report  to  the 
President  and  the  Congress  [3}  highlights  the  importance  of  military 
transformation  by  noting  that  "Transformation  lies  at  the  heart  of  our 
efforts  to  reduce  risk  posed  by  future  challenges." 

Transforming  the  U.S.  Navy 

The  Department  of  the  Navy  has  invested  substantial  intellectual  capital 
in  coming  to  grips  with  how  to  transform  the  Navy  and  the  Marine 
Corps.  The  Department  of  the  Navy’s  plans  for  transformation  were  for¬ 
mally  articulated  in  The  Naval  Transformation  Roadmap ,  released  in  July 
2002  [4].  This  document  set  a  clear  course  for  transforming  the  Navy  and 
the  Marine  Corps.  The  Chief  of  Naval  Operations’  (CNO’s)  vision  for 
Navy  Transformation,  called  Sea  Power  21,  was  articulated  in  a  series  of 
articles  beginning  in  October  2002  in  the  U.S.  Naval  Institute 
Proceedings  [5]. 

Transformation  involves  changes  in  technology,  policies,  procedures,  and 
designs  that  improve  performance  and  save  costs.  Key  tenets  of  Sea 
Power  21  focus  efforts  such  as  those  of  the  Human  Factors  Engineering 
(HFE)  consortium  at  SSC  San  Diego  and  lead  to  the  design  of  systems 
that  enable  significant  performance  gains  with  optimized  personnel. 

The  Navy  has  increased  efforts  to  capitalize  on  HFE  research  results.  For 
example,  in  the  Navy’s  Fiscal  Year  2003  N1  Playbook,  the  Chief  of  Naval 
Personnel,  Vice  Admiral  Norbert  Ryan,  notes  "The  design  of  new  sys¬ 
tems  must  include  Sailors  from  the  start"  [6]. 

Recent  fleet  studies  indicate  that  the  Navy  is  firmly  committed  to  efforts 
to  reduce  the  crew  size  on  ships.  For  example,  in  the  case  of  the  Navy’s 
CVN  21  program,  the  Navy  changed  requirements  dramatically  in  the 
fall  of  2002,  requiring  the  first  ship  of  the  new  CVN  21  class  to  have  a 
crew  size  that  is  800  less  than  the  current  Nimitz- class  carriers  [7],  The 
success  of  such  manpower  reduction  efforts  is  inextricably  linked  to  suc¬ 
cessful  HFE  during  design  and  development. 

Efforts  to  use  technology  to  reduce  manning  are  not  limited  to  new- 
construction  ships.  For  example,  the  Naval  Sea  Systems  Command  has 
commissioned  an  exhaustive  study  to  determine  ways  that  technology 
can  lead  to  reduced  manpower  requirements  on  the  Arleigh  Burke  class 
destroyers.  This  study  strongly  advocated  using  groundbreaking  Human 
Factors  Engineering  technologies  while  validating  the  need  to  reduce 
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manning  on  all  Navy  ships  by  noting  that,  since  1985,  the  Navy’s  Total 
Operating  Budget  has  declined  by  approximately  40%  and  ship  count  by 
45%;  however,  the  Operations  and  Support  (O&S)  costs  (consisting  of 
personnel,  maintenance,  consumables,  and  sustaining  support)  have 
remained  constant  during  this  time.  Personnel  costs  comprise  over  50% 
of  O&S  costs,  and  these  personnel  costs  have  been  growing  more  rapidly 
than  other  costs. 

Thus,  for  both  new-construction  ships  and  existing  ships,  platforms,  and 
command  centers,  there  is  an  ongoing  search  by  the  Navy  for  improved 
human-systems  design  and  technologies  that  enhance  human  perform¬ 
ance.  These  initiatives  have  been  formalized  in  three  key  enablers  for 
naval  transformation:  Sea  Trial,  Sea  Warrior,  and  Sea  Enterprise. 

Significantly,  Sea  Enterprise  will  use  technology  such  as  MMWS  and 
K-Web  to  empower  personnel  to  achieve  warfighting  effectiveness  in  the 
most  cost-effective  manner  [5].  The  complex  missions  undertaken  by 
naval  forces  rarely  enable  manual  processes  to  be  replaced  by  automated 
ones  with  a  "simple"  substitution  of  technology  for  operators.  Instead 
the  process  becomes  "mixed,"  with  human  supervision  of  automated 
processes  and  human  selection  of  automation  levels.  Cost  comparisons  of 
human  versus  machine  must  account  for  these  mixed-initiative  systems. 
HFE  researchers  at  SSC  San  Diego  have  studied  interaction  with  mixed- 
initiative  systems  and  developed  guidelines  to  support  effective  human- 
system  interface  design.  A  discussion  of  the  design  techniques  used  to 
support  various  levels  of  automation  is  important  to  understanding  the 
relationship  of  complex  system  design  and  crew  optimization. 

HUMAN  FACTORS  ENGINEERING:  ENHANCING  OPERATOR 
PERFORMANCE 

The  Office  of  Naval  Research  has  sponsored  research  in  Human  Factors 
Engineering  concepts  at  SSC  San  Diego  for  several  decades.  Research 
conducted  in  the  1980s  and  1990s  supported  the  recent  Multi-Modal 
Watchstation  project  and  further  progressed  into  two  Future  Naval 
Capability  (FNC)  projects  supporting  improved  Land-Attack  systems  in 
Knowledge  Superiority  and  Assurance  and  Capable  Manpower  [8]. 
Lessons  learned  can  be  summarized  into  three  major  factors:  (1) 
human-computer  interface  (HCI)  design,  (2)  information  and  software 
architecture  supporting  effective  human-computer  interaction,  and  (3) 
effective  human  factors  design  process  [9]. 

There  is  a  direct,  but  complex,  causal  link  between  effective  HFE  and 
personnel  costs.  Systems  that  are  efficient  and  easier  to  operate  require 
fewer  personnel  resources  in  all  phases  of  training  and  operation.  Poor 
design  creates  increased  personnel  burden  and  increased  risk  of  mission 
failure,  by  inducing  error  and  delays  during  peak  mission  task  loads. 

So  what  is  "effective  design,"  and  how  do  we  know  when  we  achieve  it? 
First,  cognitive  work  domain  and  task  analysis  is  a  core  part  of  the  HFE 
methodology  [10].  Effective  design  does  not,  by  its  nature,  have  to  be 
complex  or  expensive.  Sometimes  simple  solutions  produce  significant 
performance  gains  such  as  SSC  San  Diego  research  that  led  to  a  new 
method  for  selecting  objects  on  a  display  by  shifting  more  of  the  selec¬ 
tion  work  from  the  human  visual  and  motor  systems  to  the  computer 
[11].  The  resulting  changes  improved  performance  for  all  types  of  input 
devices. 
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On  a  larger  scale,  human  performance  is  transformed  through  redesign  of 
the  tactical  HCI  and  user-interactive  process  [12].  Research  results  indi¬ 
cated  significant  improvements  in  situational  awareness  and  task  response 
for  a  typical  Air  Defense  Warfare  team.  In  both  design  cases  listed  above, 
it  was  most  useful  to  start  from  a  "blank  sheet"  of  paper  and  define  criti¬ 
cal  HFE  requirements.  These  requirements  and  design  attributes  evolved 
through  research  and  testing,  and  are  related  to  a  school  of  thought  in 
HCI  design  termed  as  "Ecological  Interface"  design  [13].  This  type  of 
design  directly  reflects  and  supports  the  mission  process  and  visualization 
of  that  process.  As  illustrated  in  Figure  1,  dynamic  process  visualization 
can  be  an  important  feature  in  supporting  mission  situation  awareness. 
Tomahawk  and  Guns  reflect  step-wise  processes  while  Air  Defense  is 
range-based  and  Engine  Propulsion  is  time-based.  Visualization  supports 
important  cognitive  requirements  related  to  user  task  roles;  responsibili¬ 
ties;  past,  current,  and  future  status. 

Also,  Human  Factors  Engineering  researchers  at  SSC  San  Diego  have 
identified  a  key  requirement  that  software  functions  must  support  the 
construction  of  "mission  task  products— the  quality  of  these  products  are 
key  performance  enablers.  These  requirements  have  been  summarized 
recently  in  the  SSC  San  Diego  concept  of  a  Goal-explicit  Work  Interface 
System  (G-WIS)  [14].  The  G-WIS  is  a  representative  example  of 
"Mission-Centered  Computing"  [15].  The  G-WIS  visualization  does  not 
presume  an  "office"  Graphical  User  Interface  (GUI)  look  or  feel.  HCI 
tools  within  that  metaphor  have  been  found  sometimes  to  be  impediments 
to  the  efficient  performance  required  in  fast-reaction  weapons  systems 
[16].  Performance-enabling  properties  of  the  G-WIS  design  approach 
have  been  found  in  fleet  performance  and  usability  testing  [17].  The  sig¬ 
nificant  performance  enabler  is  not  the  HCI  look  and  feel  but  instead  the 
quality  of  the  task  products  and  their  contribution  to  the  mission 
process.  The  degree  of  impact  on  manning  and  transformation  is  directly 
related  to  the  product  quality  and  availability  across  the  gamut  of  tasks  in 
varied  mission  domains. 

The  mission  process  and  product  requirements  are  captured  through 
structured  analysis  of  workflows  and  captured  in  HFE  sequence  dia¬ 
grams  and  software  Use  Case  and  Activity  Diagrams.  Figure  2  presents 
a  typical  workflow  analysis  designed  by  HFE  researchers  at  SSC  San 
Diego.  It  shows  the  actions  of  human,  system,  and  external  entities  by 
showing  the  path  of  information  flow  and  processes.  Links  to  display 
examples  are  shown  in  the  diagram  for  viewing  the  content  of  decision 
aids  at  that  point  in  the  process  flow.  The  workflows  are  also  part  of  the 
Design  Reference  Missions,  which  contain  the  workload  and  mission 
demands  required  of  the  human-system  combination.  The  workflow 
analysis  also  reveals  mission  process  flaws  that  can  be  improved.  This 
analysis  may  include  a  reduction  of  steps  or  methods  that  may  be  unnec¬ 
essary  artifacts  from  legacy  systems.  Understanding  a  mission  process 
and  improving  it  is  critical  to  support  crew  optimization  and  naval 
transformation. 

The  complexity  of  measuring  the  impact  of  Human-System  Integration 
(HSI)  cuts  across  technology,  system  integration,  and  mission  processes 
and  protocols.  As  defined  in  a  mixed-initiative  system,  automation  is 
not  a  dichotomy  existing  in  either  an  "off"  or  "on"  state  but  is  instead  a 
continuum  across  multiple  levels  of  human  supervisory  control.  HFE 
researchers  at  SSC  San  Diego  have  shown  conclusively  that  models  cannot 
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FIGURE  1 .  Dynamic  mission  process 
visualization  displays. 
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FIGURE  2.  Example  of  workflow  analysis  linking  workflow  to  decision  support  displays. 


simply  trade  off  automation  for  human  processing  one-to-one.  Given 
the  interaction  between  design  and  process  factors,  each  factor  must  be 
included  in  models  that  estimate  design  impact  on  crew  workload  and 
crew  size.  Toward  this  end,  the  models  that  define  cost  variables  impact¬ 
ing  crew  size  and  cost  must  be  as  accurate  and  objective  as  possible. 
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MANPOWER  COST  ANALYSIS:  STILL  AN  IMPERFECT  SCIENCE 

Regardless  of  the  effectiveness  of  various  HCI  technologies,  cost  weighs 
heavily  on  strategic  decisions  regarding  technology  purchase.  Decisions 
will  be  made  within  the  context  of  hardware,  software,  and  personnel 
costs  if  these  new  systems  are  installed.  These  important  trade-offs 
should  be  made  in  an  objective  manner  with  reliable  metrics  to  guide  the 
Services  toward  the  correct  decisions. 

Strong  anecdotal  evidence  suggests  that  the  metrics  used  to  quantify  the 
cost  of  a  warfighter  provide  only  rough  approximations  of  costs.  While 
there  are  many  reasons  for  this,  an  exhaustive  study  by  the  Center  for 
Naval  Analysis  (CNA)  concluded  that,  within  the  Navy,  there  are  insuffi¬ 
cient  organizational  imperatives  to  mate  technology  and  manpower  deci¬ 
sions  [18]. 

For  example,  the  workyear  rates  promulgated  to  determine  Future  Year 
Defense  Plan  (FYDP)  requirements  for  manpower  provide  a  single  rate 
for  officers  and  a  single  rate  for  enlisted  personnel,  making  no  distinction 
among  paygrades."'  This  averaging  of  rates  skews  any  attempt  to  derive 
objective  data  regarding  personnel  costs.  This  may  tend  to  make  legacy 
systems  appear  to  be  as  cost-effective  as  new  human-systems  technologies 
by  obscuring  the  fact  that  more  junior,  less-experienced  personnel  can  be 
trained  on  new  systems  with  improved  F1SI  versus  legacy  systems  that 
required  more  experienced  operators. 

While  Navy  manpower  models  purport  to  include  all  costs  of  manpower 
(and  they  do  a  reasonable  job  of  that),  in  reality  they  quantify  that  which 
is  readily  quantifiable  while  omitting  some  important  costs  that  do 
impact  the  "life-cycle  cost"  of  personnel.  For  example,  the  Navy  "model" 
does  not  readily  factor-in  recruiting  or  training  costs,  often  obscuring  the 
extremely  long  pipeline  training  for  some  personnel  such  as  aviators  and 
nuclear- trained  officers.  The  model  is  also  not  easily  adapted  to  factor- in 
the  extraordinary  costs  of  war,  including  special  pay  for  being  in  a  war 
zone.  Additionally,  there  is  no  way  to  factor-in  the  vast  infrastructure  of 
Family  Support  Centers,  Child  Development  Centers,  and  similar  family 
support  entities. 

In  short,  while  manpower  analysts  have  done  a  credible  job  of  deriving 
a  first-order  approximation  of  Navy  manpower  costs,  institutional  factors 
auger  against  their  refining  these  metrics  to  make  it  a  precise  instrument 
to  enable  objective  manpower-technology  trade-offs.  Unless  or  until 
these  models  are  refined,  manpower  cost  analysis  will  remain  an  imperfect 
science. 


CONCLUSIONS  AND  THE  IMPERATIVES:  FURTHER  RESEARCH 

Military  transformation  will  continue  to  demand  that  technology  replace 
manpower  on  platforms,  systems,  and  command  centers.  HCI  technology 
like  MMWS  and  K-Web  developed  at  SSC  San  Diego  can  enable  mission 
processes  to  be  completed  in  a  timely  and  effective  manner  with  opti¬ 
mized  numbers  of  personnel.  Quite  often,  the  roles  of  warfighters  will 
need  to  shift  toward  supervisory  control  of  multiple  mission  processes 
versus  manual  control  of  a  single  mission  process.  Software  systems  must 
be  designed  to  produce  high-quality  mission  products  enabling  effective 
warfighting  with  fewer  personnel.  HFE  researchers  at  SSC  San  Diego 


"'Chief  of  Naval  Operations  (N10)  directive  dated  22  January  2003,  Serial 
N1021/01.  In  January  2003,  the  FY  03  "cost"  for  an  officer  (O-l  to  0-10)  was 
$100,106,  and  the  "cost"  for  a  sailor  (E-l  to  E-9)  was  $49,619. 


have  shown  that,  in  many  cases,  costs  for  duplicate  functionality  can  be 
shared  across  systems,  thereby  reducing  the  cost  of  automation  or  deci¬ 
sion  support.  The  costs  of  better  automation  and  high-quality  software 
mission  products  must  be  compared  to  the  "true"  cost  of  personnel. 

Directly  comparing  the  manpower  costs  to  systems  development  and 
maintenance  costs  does  not  always  tell  the  entire  story,  nor  does  it 
necessarily  provide  a  complete  and  objective  analysis.  The  quality  and 
reliability  of  performance,  coupled  with  the  speed,  accuracy,  and  efficiency 
of  decision-making  ultimately  impact  the  mission  performance  of  these 
operators.  Clearly,  this  is  an  area  requiring  more  research  and  modeling 
to  determine  the  viability  of  coordinating  the  optimal  mix  of  smarter 
systems  and  crew  size.  This  area  also  demands  research  that  will  lead  to 
more  effectively  defining  the  "true  cost"  of  an  officer  or  an  enlisted  per¬ 
son  on  Navy  ships,  and  this  research  can  reduce  the  risk  that  "manpower- 
heavy"  Navy  platforms  will  become  unaffordable,  inhibiting  Department 
of  the  Navy  transformational  initiatives  and  reducing  the  contribution 
that  the  Navy  can  make  to  the  National  Security  of  the  United  States. 
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INTRODUCTION 

Tactical  sensor  employment  is  hard  to  learn  and  hard  to  train.  Extensive 
knowledge  and  substantial  inferential  capability  are  required  to  interpret 
sensor  data,  generate  hypotheses  about  their  meaning,  and  propose 
courses  of  action.  In  the  real  world,  the  proper  interpretation  and  use  of 
sensor  data  are  among  the  most  difficult  tasks  in  many  science-based 
fields  of  endeavor.  Examples  include  use  of  geological  data  in  oil  explo¬ 
ration,  imagery  and  biochemical  test  results  in  medical  diagnosis, 
spectrographic  and  bio-chemical  data  in  forensic  analysis,  and  electro¬ 
magnetic,  electro-optic,  and  acoustic  signal  analysis  in  naval  warfare. 

All  of  these  tasks  require  deep  understanding  of  the  physical  properties 
being  sensed,  the  operation  and  limitations  of  sensors,  and  the  environ¬ 
mental  or  real-world  interactions  that  affect  data  observation  and  inter¬ 
pretation.  In  most  warfare  applications,  these  tasks  are  even  more 
difficult  because  intelligent  opponents  seek  to  avoid  detection,  confuse 
identification,  and  gain  tactical  advantage  by  employing  intelligent 
countermeasures  or  unconventional  maneuvers. 

In  many  fields,  graduate-level  degrees  and  many  years  of  experience  are 
required  to  develop  the  necessary  skills  for  reasoning  in  dynamic,  highly 
variable,  and  ambiguous  situations.  In  contrast,  junior  enlisted  operators 
in  the  Navy  often  perform  sensor  employment  tasks  that  are  essential  to 
the  tactical  outcome  of  combat  events.  Most  operators  are  high-school 
graduates  with  very  limited  formal  education  in  physics  and  engineering, 
and  with  no  experience  in  real-world  operations  with  non-cooperative 
opponents. 

In  antisubmarine  warfare  (ASW),  the  training  challenges  are  especially 
formidable.  Foreign  nuclear  submarine  technology  continues  to  improve, 
and  advanced  submarines  continue  to  be  built  and  delivered.  At  the  same 
time,  the  proliferation  of  improved  diesel-submarine  technology  to  many 
Third  World  nations  requires  that  ASW  forces  adapt  to  those  threats  as 
well.  Since  ASW  is  no  longer  only  an  open-ocean,  deep-water  enterprise, 
operations  must  now  be  conducted  in  the  vastly  different  littoral  regions. 
Also,  in  the  past,  extensive  opportunistic  practice  occurred  in  the  normal 
course  of  submarine  operations.  Today,  because  contact  opportunities 
with  capable  potential  adversaries  are  relatively  infrequent,  new  training 
is  necessary  to  develop  the  knowledge  required  to  deal  with  quieter  tar¬ 
gets  in  more  complex  environments,  and  more  practice  opportunities  are 
necessary  to  develop  advanced  sensor  employment  and  tactical  skills  that 
were  previously  learned  on  the  job. 


ABSTRACT 

The  objective  of  the 
Interactive  Multisensor 
Analysis  Training  (IMAT) 
project  is  to  better  train 
operational  users  of  undersea- 
warfare  sensor  systems.  The 
effort  has  focused  on  training 
at  all  levels  from  initial  train¬ 
ing  ashore  through  team,  plat¬ 
form,  and  collective  training 
at-sea,  at  all  skill  levels  from 
apprentice  sensor  operators  to 
senior  tactical  commanders. 
Operators  and  tacticians  at  all 
levels  need  a  deep  and  scientif¬ 
ically  accurate,  but  not  neces¬ 
sarily  formal  understanding  of 
the  physical  principles  that 
underlie  tactical  use  of  sensor 
systems.  IMAT  systems  use 
scientific  visualizations,  three- 
dimensional  graphics,  and 
animations  to  illustrate  com¬ 
plex  physical  interactions  in 
mission-relevant  contexts. 
Instruction  concepts  include 
radiated  acoustic  characteris¬ 
tics,  propagation  in  range- 
dependent  environments,  and 
sensor  properties.  Training  sys¬ 
tems  provide  exploratory  envi¬ 
ronments  in  which  operators 
and  tacticians  can  examine  the 
effects  of  change  in  any  of  the 
variables  involved  in  the  end- 
to-end  sequence  of  emission, 
transmission,  reflection,  and 
detection.  Sensor  settings,  envi¬ 
ronmental  conditions,  and  tar¬ 
get  characteristics  can  all  be 
modified  through  a  "what-if" 
simulation  approach.  These 
technologies  have  been  applied 
effectively  in  basic  and  ad¬ 
vanced  sensor  operations/ 
employment  courses, 
in  individual  and  team 
training  simulators,  and 
in  onboard  training. 
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THE  IMAT  PROGRAM 

The  Interactive  Multisensor  Analysis  Training  (IMAT)  program  is  pro¬ 
viding  training  and  performance  support  systems  designed  to  make  diffi¬ 
cult  scientific  and  technical  concepts  comprehensible  to  the  operational 
users  of  advanced  sensor  systems.  IMAT  goals  include  (1)  developing 
systems  that  integrate  computer  models  of  physical  phenomena  with 
scientific  visualization  technologies  to  demonstrate  the  interactive  relation¬ 
ships  of  threat,  environment,  and  sensor  for  operator  training,  and  inter¬ 
actions  of  multiple  sensor  systems  for  tactician  training;  (2)  developing 
training  and  performance  support  systems  using  modeling  and  visualiza¬ 
tion  technologies;  (3)  integrating  curricula  to  provide  training  on  high- 
level  sensor  operation  and  tactical  planning  skills;  and  (4)  developing 
modeling  and  visualization  tools  for  use  at  sea,  both  for  training  and  as 
tactical  decision  aids. 

Several  IMAT  visualization  systems  have  been  developed.  A  large-scale 
version  of  the  system  uses  high-end  workstations  with  special-purpose 
parallel  processing  to  provide  very  rapid  sensor  performance  visualiza¬ 
tion.  This  version  is  used  primarily  as  a  tool  for  exploring  tactical  impli¬ 
cations  of  sensor  employment,  and  also  as  an  instructor  tool  in  classroom 
settings.  This  system  is  also  the  basis  for  new-technology  multi-operator 
submarine  sonar  training  systems,  and  for  deployed  training  and  tactical 
visualization  on  submarines  and  surface  ships.  Personal  computer  (PC) 
IMAT  is  a  laptop-based  system  that  can  be  used  both  tactically  and  as  an 
instructional  tool.  The  system  allows  individual  users  to  make  timely 
sensor  performance  predictions  based  on  available  environmental  data, 
and  it  allows  students  to  review,  reinforce,  and  explore,  at  their  own  pace, 
complex  concepts  involved  in  ASW.  The  system  has  recently  been 
extended  to  provide  shared  operational  information  over  afloat  networks 
for  collaborative  tactical  planning,  multi-platform  battle-group-level 
situation  assessment  and  analysis,  and  distributed  training. 

This  paper  briefly  describes  how  basic  concepts  are  taught  using  IMAT  in 
apprentice  sonar-operator  courses.  These  basic  concepts  include  funda¬ 
mentals  of  acoustics,  acoustic  properties  of  targets,  properties  of  sensors, 
and  environmental  effects  on  propagation.  The  approaches  for  simulation- 
based  sonar  training  and  at-sea  training  at  the  ship  and  battle-group  levels 
are  also  discussed. 

Acoustics 

Basic  concepts  of  sound  and  wave  theory,  such  as  sound  pressure,  fre¬ 
quency,  wavelength,  velocity,  and  amplitude  are  initially  introduced  by 
using  visualizations.  The  intent  is  to  give  qualitative  explanations  of  the 
underlying  phenomena.  From  the  beginning,  these  topics  are  taught  in  a 
"real-world"  context  by  relating  them  to  properties  of  submarines  rele¬ 
vant  to  the  tasks  of  detection,  localization,  and  classification. 

The  submarine  display  (Figure  1)  provides  animated  visualizations  of  the 
internal  components  of  a  submarine.  The  display  is  linked  to  recordings 
of  actual  acoustic  data.  Selecting  a  motor,  pump,  or  other  object  will  dis¬ 
play  a  description  and  will  highlight  frequency  lines  associated  with  the 
component  on  the  sonogram  in  the  bottom  part  of  the  display.  This 
enables  the  student  to  better  understand  how  complex  assemblies  work, 
why  they  generate  certain  signals,  and  how  signals  relate  to  operating 
mode  and  speed.  Selectable  components  include  examples  of  diesel 
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engines,  turbines,  reduction  gears, 
pumps,  propellers,  motors,  gener¬ 
ators,  compressors,  and  blowers. 

In  addition,  a  high-fidelity  acoustic 
simulator  is  also  included.  Most 
parameters  that  control  the  simu¬ 
lation  can  be  varied  and  explored 
for  instructional  purposes. 

Sensors 

The  properties  and  functions  of 
acoustic  sensors  are  explained  in 
the  context  of  detection  and  local¬ 
ization,  that  is,  how  the  sensitivity 
of  sensors  can  be  directed  so  as  to 
increase  signal  relative  to  noise, 
and  so  as  to  provide  directional 
information.  Again,  interactive 
animations  are  used  to  explain 
underlying  concepts.  For  exam¬ 
ple,  for  principles  of  beamforming 

using  a  phased  array,  a  three-  .  ,  ,  .  . 

...  ,  ,  ,  .  r  FIGURE  1 .  Power-tram  components  related  to  acoustic  signature, 

dimensional  rendered  view  ot 

isosensitivity  was  provided  for  a 

given  combination  of  array  elements,  inter-element  spacing,  and  phase 
delays.  The  system  can  accommodate  multi-aperture  and  other  (e.g.,  non¬ 
linear)  array  geometries.  The  user  can  vary  all  parameters  in  these  dis¬ 
plays  in  order  to  investigate  beam  width  and  directivity  as  a  function  of 
array  design  and  employment. 

Environment 

A  third  part  of  the  conceptual  foundation  for  sensor  employment 
involves  environmental  effects  on  sound  transmission  in  the  ocean.  IMAT 
includes  an  interactive  modeling  facility  to  help  students  explore  and 
understand  transmission  loss.  Since  transmission  loss  is  affected  by 
spreading,  absorption  by  the  bottom,  and  scattering  at  the  bottom  and 
surface,  all  these  factors  are  controllable  in  the  interactive  displays.  IMAT 
includes  extensive  environmental  models,  including  Parabolic  Equation 
(PE)  and  Gaussian  Ray  Bundle  (GRAB)  range-dependent  propagation 
loss  models,  and  databases  on  bathythermography,  bottom  absorption, 
and  other  oceanographic  data,  all  approved  by  the  Oceanographer  of  the 
Navy.  With  these  modules,  a  user  can  select  any  geographic  location  and 
time  of  year,  extract,  view,  enter,  and/or  modify  environmental  data  (such 
as  sound  speed,  bottom  loss  or  reflectivity),  enter/modify  source  and  tar¬ 
get  depths  and  frequency  of  interest,  and  then  model  propagation  loss  as 
a  function  of  depth,  distance,  and  azimuth  from  a  sensor  or  threat.  Figure 
2  shows  an  example  full-field  plot,  with  the  bottom  panel  showing  trans¬ 
mission  loss  over  range  at  the  indicated  sensor  depth.  At  a  very  early 
point  in  training,  students  can  be  introduced  to  the  high  degree  of  vari¬ 
ability  in  environmental  features  and  to  its  importance  in  sensor  employ¬ 
ment. 

Many  other  interactive  displays  for  conceptual  understanding  are  avail¬ 
able  in  IMAT  and  PC-IMAT,  including  properties  of  other  sensors  such 
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as  sonobuoys,  electro-magnetic 
and  electro-optical  sensor  sys¬ 
tems,  and  basic  and  advanced 
concepts  for  active  systems, 
including  multistatics. 

TRAINING  EFFECTIVENESS 

IMAT  shore-school  products 
have  transitioned  from  research 
and  development  and  are  now 
used  in  over  20  apprentice-to- 
advanced  courses  in  the  submarine, 
surface,  air,  and  meteorological/ 
oceanographic  communities. 
Evaluations  of  training  effective¬ 
ness  in  shore  schools  indicate 
that  IMAT  is  among  the  most 
successful  training  technologies 
ever  introduced  in  the  Navy. 

The  Naval  Studies  Board  of  the 
National  Academy  of  Sciences  [1] 
noted: 
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FIGURE  2.  Full-field  transmission  loss  plot. 


•  IMAT  students  outperform 
students  in  conventional 
instruction,  and,  in  many  cases, 

score  higher  than  qualified  fleet  personnel  with  3  to  10  years  experi¬ 
ence.  Evaluations  consistently  show  gains  of  two  to  three  standard 
deviations  on  comprehension,  reasoning,  and  problem-solving  tasks. 
Overall,  the  IMAT  approach  is  much  more  effective  than  conventional 
lecture  instruction  or  new  technologies  such  as  interactive  video  or 
computer-based  training. 

•  Instructors  report  that  IMAT  increases  their  ability  to  teach  difficult 
topics,  respond  to  student  questions,  and  reinforce  critical  principles. 

•  IMAT  students  score  higher  on  attitude  scales  measuring  attention, 
relevance,  confidence,  and  satisfaction  than  students  in  standard  Navy 
classrooms  or  students  in  specially  designed  individualized  computer- 
based  training. 

•  IMAT  development  costs  for  initial  courses  are  equivalent  to  or  less 
than  conventional  courses  and  less  expensive  than  other  new-technology 
courses.  Subsequent  development  of  related  training  is  up  to  90% 

less  expensive. 

The  Commander,  Naval  Education  and  Training  Command  and  the 
Office  of  the  Chief  of  Naval  Operations  (OPNAV)  sponsors  are  currently 
completing  implementations  throughout  the  submarine  and  air  ASW 
training  pipelines  and  are  planning  for  expansion  of  IMAT  training  in  the 
surface  community. 


TEAM  TRAINING  FOR  TACTICAL  SENSOR  EMPLOYMENT 

IMAT  technologies  support  more  advanced  team-  and  platform-level 
training  in  tactical  employment  of  sensors  and  tactical  use  of  the  environ¬ 
ment.  A  new-technology  submarine  sonar  employment  trainer  (SET), 
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which  uses  IMAT  visualizations,  is  now  being  delivered  to  the  Naval 
Submarine  School.  The  primary  functions  of  the  SET  are  to  provide 
instructor-controlled,  scenario-based  training  with  "what-if "  capabilities. 
This  training  will  support  development  of  reasoning  concerning  sonar 
systems  employment  and  tactics  by  exposing  trainees  to  experiences  that 
might  only  have  been  encountered  opportunistically  during  mission 
deployments.  These  scenarios  will  allow  sonar  operators,  sonar  super¬ 
visors,  and  sonar  coordinators  to  work  as  if  deployed,  and  to  explore 
alternative  courses  of  action  to  maximize  learning  from  a  variety  of 
operational  situations  and  environmental  conditions. 

This  work  has  also  transitioned  to  the  Submarine  Multi-Mission  Team 
Trainer,  Phase  3  (SMMTT3).  The  SMMTT  is  a  full  combat  systems  team 
trainer  for  the  sonar,  fire-control,  and  combat-center  teams.  Acquisition 
funding  began  in  fiscal  year  (FY)  02  and  is  programmed  through  FY  06 
for  systems  at  all  six  submarine  training  facilities. 


AT-SEA  PERFORMANCE  SUPPORT 


More  recent  work  has  extended  IMAT  technologies  to  onboard  mission 
support  for  ASW  operations.  For  more  complicated  tactical  analyses, 
precise  data  are  necessary  for  entry  into  sensor-performance  predictions. 
Therefore,  extensive  databases  on  threat  characteristics  and  sensor  sensi¬ 
tivity/directivity  are  included.  These,  together  with  the  high-resolution 
environmental  databases  described  previously,  allow  users  to  do  exercise 
and  mission  planning,  execution  monitoring,  and  reconstruction.  IMAT 
systems  provide  visualization  tools  for  both  monitoring  of  a  current 
tactical  situation,  as  well  as  for  investigating  "what  if"  suppositions. 
IMAT/PC-IMAT  systems  are  approved  as  Tactical  Decision  Aids  on  all 
U.S.  submarines  and  surface  combatants.  IMAT  visualizations  are  now 
part  of  the  submarine  acoustic  rapid  commercial-off-the-shelf  (COTS) 
insertion  (ARCI)  combat  systems  acquisition,  and  are  Program  of  Record 
Tactical  Decision  Aids  for  submarine,  surface-ship,  and  Integrated 
Undersea  Surveillance  System  (IUSS)  applications. 


The  PC-IMAT  Destroyer  Squadron 
recently  been  developed  to  pro¬ 
vide  battle-group-level  planning 
and  monitoring  tools.  Figure  3 
shows  the  installation  aboard  USS 
Kitty  Hawk  (CV  63).  The  system 
is  also  capable  of  interacting  over 
secure  networks  with  other  PC- 
IMAT  systems  to  share  data  and 
to  provide  a  common  tactical  pic¬ 
ture.  Further  work  is  currently 
integrating  the  system  with  other 
ASW  tactical  systems  to  develop 
the  Common  Undersea  Picture. 

The  overall  program  is  developing 
battle-group-  and  theater-level 
visualization  systems  to  support 
multiplatform  ASW  tactical 
planning,  tactical  support,  and 
reconstruction/feedback. 


(DESRON)  Support  System  has 


FIGURE  3.  PC-IMAT  DESRON  Support  System,  USS  Kitty  Hawk. 
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IMAT  FLEET  TRAINING 

Since  1997,  the  IMAT  program  has  also  been  developing  new  approaches 
to  waterfront  and  onboard  training.  The  strategy  has  been  to  provide 
extensive  pre-exercise  training  to  the  sonar  and  combat  teams,  during 
which  the  upcoming  exercise  is  used  as  a  basis  for  tactical  planning.  Then, 
IMAT  project  personnel  provide  at-sea  training  and  support  during  the 
exercise  and  also  provide  reconstructions  and  lessons  learned.  To  date, 
this  effort  has  supported  10  major  carrier  battle-group  workup,  exercise, 
and  deployment  cycles.  In  2002,  the  Commander,  Pacific  Fleet  (COM- 
PACFLT)  and  Fleet  Forces  Command  institutionalized  this  effort  as  a 
Fleet  Training  Program  of  Record.  IMAT  fleet  training  is  a  central  part  of 
the  COMPACFLT  initiative  to  re-invigorate  ASW  proficiency. 


CONCLUSION 

Today,  IMAT  is  a  unique  set  of  efforts  that  cross  sponsor,  mission,  plat¬ 
form,  and  sensor  boundaries.  IMAT  is  truly  interdisciplinary,  including 
work  in  physical  acoustics;  oceanography;  electromagnetics  and  electro¬ 
optics;  modeling  and  simulation;  training  and  information  management 
technologies;  sensors,  processors,  and  display  technologies;  tactics  devel¬ 
opment  and  analysis;  environmental  data  collection  and  distribution;  and 
command,  control,  communications,  computers,  and  intelligence  (C^I). 

The  IMAT  vision  is  to  integrate  training,  operational  preparation,  tactical 
execution,  and  post-mission  analysis  into  a  seamless  support  system  for 
developing  and  maintaining  mission-related  critical  skills.  In  many  ways, 
IMAT  is  a  prototype  for  future  human  performance  support  systems  that 
transcend  traditional  shore  school  and  course  structures  to  span  career- 
long  skill  development  from  apprentice  to  master  levels,  across  missions, 
platforms,  and  communities. 
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INTRODUCTION 

The  U.S.  Central  Command  Deployable  Headquarters  (CDHQ),  now 
referred  to  as  Command  Headquarters  Forward  or  CHF,  was  a  rapid 
acquisition  build  for  deployment  to  Qatar  in  the  Persian  Gulf.  The 
CDHQ  provides  a  deployable  headquarters  to  the  joint  service  on-scene 
commander.  Included  in  the  CDHQ  are  state-of-the-art  communications 
and  a  command,  control,  communications,  computers,  and  intelligence 
(C4I)  application  computing  infrastructure.  The  CDHQ  was  produced 
in  less  than  10  months  by  a  unified  team  made  up  from  multiple  services, 
civilian  agencies,  and  companies.  SSC  San  Diego  provided  leadership  and 
engineering  for  the  system  architecture,  systems  design,  applications 
integration,  integrated  logistics,  and  safety,  as  well  as  communications 
and  security  for  the  CDHQ.  When  completed,  the  CDHQ  provided  a 
deployable  headquarters  for  over  250  warfighters.  Deployed  to  Qatar  in 
the  Persian  Gulf,  the  first  major  test  of  CDHQ  was  during  the  Internal 
Look  ’03  exercise  in  December  2002,  followed  by  Operation  Iraqi 
Freedom  and  continuing  operations. 

The  CDHQ  was  a  large-scale  systems  development  and  integration 
effort  performed  under  highly  constrained,  poorly  defined  conditions.  In 
this  paper,  we  describe  SSC  San  Diego’s  efforts  as  part  of  the  CDHQ 
team  and  the  lessons  learned  in  the  CDHQ’s  production  and  delivery. 

BACKGROUND 

Prior  to  11  September  2001,  planning  had  begun  for  the  Deployable 
Headquarters  (DHQ)  Advanced  Concept  Technology  Demonstration 
(ACTD).  The  Joint  Precision  Strike  Demonstration  (JPSD)  Project 
Office  and  SSC  San  Diego  proposed  a  3-year  development  program  to 
produce  the  DHQ.  After  11  September,  the  task  changed.  U.S.  Central 
Command  needed  a  forward  command  center  capability  within  months. 
The  ACTD  became  a  rapid  acquisition  program  of  the  new  U.S.  Central 
Command  Deployable  Headquarters  (CDHQ)  for  the  Commander, 
General  Tommy  Franks.  JPSD  was  chosen  as  the  program  manager  and 
SSC  San  Diego  accepted  the  task  to  lead  a  government  technology  team 
to  design  and  build  the  CDHQ.  On  25  September  2001,  an  ad  hoc  team 
convened  in  Washington,  D.C.  with  no  plans  for  funding,  specifications, 
or  design.  JPSD  issued  a  contract  to  their  prime  contractor,  Raytheon, 
who  subcontracted  to  General  Dynamics  and  others  to  deliver  the 
CDHQ.  An  interim  design  process  put  the  program  in  place  by  October 
and  a  design  concept  was  delivered  on  1  November  2001,  about  the  time 
the  first  funding  was  released. 


ABSTRACT 

SSC  San  Diego  participated  in 
the  rapid  acquisition  build  of 
the  U.S.  Central  Command 
Deployable  Headquarters 
(CDHQ)  for  deployment  to 
Qatar  in  the  Persian  Gulf. 
Originally  planned  as  an 
Advanced  Concept  Technology 
Demonstration  to  be  built  over 
a  3-year  period,  the  CDHQ 
was  produced  in  less  than  10 
months  by  a  unified  team 
made  up  from  multiple  servic¬ 
es,  civilian  agencies,  and  pri¬ 
vate  companies.  The  CDHQ 
provides  a  deployable  head¬ 
quarters  to  the  joint  service 
on-scene  commander.  Included 
in  the  CDHQ  are  state-of-the- 
art  communications  and  a 
command,  control,  communi¬ 
cations,  computers,  and  intelli¬ 
gence  (Cl I)  application  com¬ 
puting  infrastructure.  This 
paper  describes  SSC  San  Diego 
efforts  as  part  of  the  CDHQ 
team  and  the  lessons  learned  in 
its  production  and  delivery. 
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The  highest  level  requirement  was  often  summarized  by  U.S.  Central 
Command  (CENTCOM)  as  "We  want  the  C4I  capability  we  have  at 
USCENTCOM,  MacDill  AFB,  put  in  hard-sided  shelters  and  made 
ready  for  the  field."  The  CDHQ  team  determined  the  overarching 
requirements,  either  specified  or  derived  from  good  engineering  as  fol¬ 
lows.  The  CDHQ: 

1.  Shall  be  modular,  scalable,  tailorable,  field  maintainable,  and  field 
supportable. 

2.  Shall  be  deployable  via  air  (C-17  or  C-5  aircraft,  but  not  C-130s) 
or  sea,  and  ground  transportable  to  the  operational  site. 

3.  Shall  be  interoperable,  within  security  limitations,  between  joint 
and  coalition  government  organizations,  and  non-government 
organizations. 

4.  Shall  provide  approximately  250  watch  positions  as  specified  at  the 
CENTCOM  J-code,  watch  position  description,  and  by  shelter  type 
and  seating-level  charts. 

5.  Shall  provide  the  C4I  applications  capability  available  at  CENTCOM, 
plus  collaboration  capabilities. 

6.  Shall  provide  all  communications  through  the  Joint  Communi¬ 
cations  Support  Element  (JCSE),  the  designated  communications 
provider  to  CENTCOM  in  the  field. 

7.  Shall  provide  some  level  of  chemical  and  biological  protection. 

8.  Shall  be  robust  and  designed  for  future  growth  and  technology 
insertion. 

9.  Shall  be  able  to  run  off  of  50-Hz  or  60-Hz  power  from  either  com¬ 
mercial  power  or  generators. 

Between  December  and  August,  the  team  designed,  fabricated,  tested,  and 
delivered  the  deployable  headquarters  to  CENTCOM  from  the  Raytheon 
facility  in  Florida.  The  CDHQ  development  site  was  composed  of  an 
outer  Secret-level  compound  and  an  inner  sensitive  compartmented  infor¬ 
mation  facility  (SCIF),  or  Top  Secret-level  compound.  Standard  16-person 
shelters  were  designated  for  a  specified  J-Code  (J 2,  J3,  J4,  J5,  and  J6)  as 
shown  in  Figure  1,  a  specific  command  function  (i.e.,  Joint  Operations 
Center  [as  shown  in  Figure  2],  Command  Briefing  Room,  Commander  in 
Chief  and  Deputy  Commander  in  Chief  shelters 
[now  Commander  and  Deputy  Commander],  war 
room,  Theatre  Communications  Control  Center),  a 
support  function  (six  server  shelters  supporting  the 
four  networks,  NIPRNET  [unclassified  but  sensi¬ 
tive  Internet  protocol  network],  Combined 
Enterprise  Regional  Information  Exchange  System 
[CENTRIXS]  coalition  network,  SIPRNET  [secret 
Internet  protocol  network],  and  J2  JWICS  Top 
Secret  network),  or  storage. 

DESIGN  APPROACH 

The  CDHQ  was  developed  under  extreme  condi¬ 
tions.  This  section  examines  best  systems  engineer¬ 
ing  practices  under  these  conditions.  The  team’s 
focus  was  on  risk  reduction,  time-to-product,  and 
quality  of  product.  Representative  issues  and  decisions  are  discussed. 
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Refinement  of  Requirements 

The  CDHQ  program  lacked  well-defined  require¬ 
ments,  complicated  by  limited  access  to  CENT- 
COM  personnel  and  facilities  because  of  Operation 
Enduring  Freedom,  the  war  in  Afghanistan. 

Requirements  were  often  given  by  personnel  that 
were  several  layers  removed  from  the  operational 
and  support  personnel  (both  military  and  contrac¬ 
tors).  Specification  of  requirements  at  the  level  need¬ 
ed  for  a  complete  design  were  lacking,  so  the  team 
proceeded  based  on  best  engineering  processes  and 
approaches  that  left  the  maximum  dynamics  in  the 
system  for  future  changes. 

Design  Approach:  Using  the  team’s  past  experience 
and  professional  contacts,  material  received  from 

CENTCOM  was  consolidated  and  examined,  and  a 
i  r  •  -ii  i  -ii  FIGURE  2.  Joint  Operations  Center  shown  during  first  operational  use. 

plan  ior  a  generic,  sustainable,  and  survivabie  capa¬ 
bility  was  developed.  When  presenting  various 

design  approaches  and  procurements  to  the  customer,  we  would  some¬ 
times  get  back  information  such  as  "we  don’t  use  those  routers,  Cisco®  is 
what  we  are  trained  for,"  "we  don’t  use  that  software  version,  we  use  dif¬ 
ferent  versions,"  and  "we  can’t  use  that  equipment  because  we  have 
found  that  is  not  field  maintainable. "  Complications  arose  in  that  the  J2 
and  J6  communications  and  applications  requirements  were  quite  differ¬ 
ent,  and  there  were  nuances  to  the  four  networks  we  needed  to  support. 

Eventually,  we  achieved  a  compromise  between  what  the  user  wanted  and 
what  we  were  able  to  deliver  in  the  time  given. 

Reduce  Project  Complexity  and  Time-to-Build 

The  level  of  complexity  of  this  project  and  the  ambitious  delivery  schedule 
meant  we  had  to  use  some  best  practice  concepts  to  transform  our  generic 
capability  into  a  delivered  system  that  would  support  the  warfighter. 

Time-to-build  was  hampered  by  long  lead-time  items  and  the  large  num¬ 
ber  of  hardware  pieces  involved,  complicated  by  the  incremental  funding 
from  the  government. 

Design  Approach:  To  reduce  time-to-build,  we  divided  the  project  into 
several  parallel  efforts.  We  limited  our  hardware  platforms  to  only  a  few 
hardware  types  to  minimize  the  integration  problem.  By  choosing  modu¬ 
lar  components  and  following  standards,  we  also  simplified  the  work.  For 
example,  needing  to  reduce  complexity,  only  the  CUSeeMe™  Servers  and 
NetMeeting®  clients  of  the  Department  of  Defense  (DoD)  Collaboration 
Toolset  (DCTS)  were  included.  The  CDE1Q  delivery  schedule  was  creat¬ 
ed  and  the  longest  lead  times  examined  to  determine  the  best  use  of  paral¬ 
lel  efforts  (e.g.,  establishment  of  two  shelter  refurbish  and  modification 
sites  and  multiple  cable  production  sites)  and  schedule  purchases.  Initial 
purchases  of  long  lead-time  items,  mostly  hardware  such  as  computers, 
had  to  be  made  before  full  knowledge  of  requirements  and  user  prefer¬ 
ences  were  known. 

Increase  CDHQ  Field  Robustness,  Maintainability,  and  Supportability 

The  level  of  robustness,  maintainability,  and  supportability  while  fielded 
needed  to  be  better  than  that  at  CENTCOM.  In  the  communications  and 
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applications  areas,  this  was  because  of  the  lack  of  specific  vendors  and 
parts  expected  in  the  CENTCOM  area  of  responsibility.  Additional 
dynamics  were  required;  for  example,  users  needed  to  be  able  to  leave 
one  shelter  with  a  laptop,  move  to  another  shelter,  and  continue  to  work. 
There  was  little  or  no  time  for  the  CENTCOM  team  to  learn  new  systems. 

Design  Approach:  These  goals  were  met  by  use  of  modularity,  redundancy, 
dynamics,  standards,  and  commonality  wherever  possible.  Shelters  were 
designed  to  be  modular  so  that  an  application  type  shelter  could  be 
exchanged  for  another  application  type  shelter  if  required.  Redundancy 
was  built  into  the  systems;  for  example,  both  primary  and  secondary 
applications  and  communications  servers,  switches,  and  routers  were  used 
with  fail-over.  Eligh-availability  components  were  used  where  possible. 

Simplify  Security  Requirements  and  Documentation 

Simplifying  security  requirements  and  documentation  is  crucial  to  the 
delivery  of  any  major  product.  Security  must  be  considered  from  the 
beginning. 

Design  Approach:  Because  the  networks  and  computers  were  to  be  distrib¬ 
uted  throughout  the  compound,  we  decided  to  use  gigabit  fiber  to  deliver 
network  connectivity  to  each  shelter.  By  putting  all  data  and  multimedia 
communications  over  fiber,  cross-talk  issues  were  eliminated.  CENT¬ 
COM  required  that  network  cables  be  color-coded  by  classification  to 
help  ensure  proper  connection  of  networks.  We  also  decided  that  we 
would  stream  all  data  over  Internet  protocol  (IP).  All  transmitting  anten¬ 
nas  were  placed  on  the  outside  of  the  compound  for  TEMPEST 
(Transient  Electromagnetic  Pulse  Emanation  Standard)  reasons.  Security 
expertise  was  brought  in  early  to  support  development  decisions,  address 
standards,  and  develop  the  required  documentation.  Early  decisions 
greatly  affect  the  amount  of  rework  and  time-to-signoff  for  acceptance. 

Develop  CDHQ  Development  Site  and  CDHQ  Field  Site  Requirements 

A  site  layout  was  needed  to  meet  the  communications,  power  and 
grounding,  shelter  staging  dynamics,  and  security  requirements  to  estab¬ 
lish  the  temporary  SCIF  at  the  development  site  and  to  support  the  field 
deployment  site(s). 

Design  Approach:  A  notional  site  layout  was  designed  and  used  for  devel¬ 
opment  because  the  actual  initial  site  was  unknown.  The  layout  included 
approaches  that  were  dynamic  where  possible,  such  as  standard  power 
and  tactical  fiber-optic  cable  lengths  for  the  site. 

Develop  Shelter  Requirements 

The  shelters  were  standard  military  (ISO,  NAVAIR,  and  Modular 
Extendable  Rigid  Wall  Shelters  [MERWS])  honeycombed  aluminum  shel¬ 
ters  that  were  government-furnished  equipment  (GFE)  procurements. 
These  shelters  had  to  be  retrofitted  to  support  the  CDHQ  because  a  clean 
cable  plant  and  support  for  equipment  racks  were  required  for  safety  rea¬ 
sons.  Maximum  weight  requirements  for  the  shelters  also  had  to  be  met. 
To  make  them  rapidly  deployable,  each  shelter  had  either  racks  or  transit 
cases  containing  the  communications  gear,  uninterruptible  power  supplies, 
and  cable  plants. 

Design  Approach:  The  plan  was  to  make  as  many  of  the  shelters  as  identi¬ 
cal  as  possible.  Power,  communications,  and  cabling  solutions  were  stan¬ 
dardized  across  the  maximum  number  of  shelters.  Weight  planning  was 
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refined  to  ensure  shelters  were  under  their  maximum  weight.  As  part  of 
meeting  the  security  and  usability  requirements,  color-coded  cables  and 
unique  connector  types  matched  to  type  of  service  were  used.  Furniture 
and  electronics  were  also  standardized.  All  non-server  shelters  were  capa¬ 
ble  of  all  available  types  of  service,  allowing  a  change  of  usage  for  future 
operations. 

LESSONS  LEARNED 

This  section  describes  some  of  the  insights  gained  and  examples  of  prob¬ 
lems  encountered  in  the  production  of  the  CDHQ. 

Overarching  Lessons  Learned 

•  To  make  quick,  quality  decisions,  three-way  partnerships  (user,  con¬ 
tractor,  and  government)  at  all  decision  levels  were  required  to  speed 
up  the  decision  process  and  take  advantage  of  team  knowledge  and 
experience. 

•  A  consistent  understanding  of  the  constraints  of  the  program  was 
important.  A  constraint  that  we  missed  initially  is  that  software  con¬ 
strains  hardware.  Users  require  software  to  perform  their  job.  Once 
you  know  what  software  you  have  to  use,  you  can  determine  what 
hardware  you  are  "allowed"  to  use. 

•  Make  sure  you  have  enough  people  on  the  team  from  the  beginning. 
Adding  people  late  in  a  project  is  difficult,  and  the  ramp-up  time 
becomes  expensive  and  counter-productive.  This  project  was  as  suc¬ 
cessful  as  it  was  largely  because  of  "heroes"  on  the  team.  Working 
excessive  hours  every  day  to  meet  a  nearly  impossible  schedule  can 
lead  to  costly  mistakes,  affect  safety,  and  harm  team  morale. 

•  Do  not  increase  the  security  posture  of  the  work  site  too  early. 
Increasing  the  security  posture,  from  Unclassified  to  Secret,  and  Secret 
to  SCI/Top  Secret,  too  early  caused  unnecessary  hindrance  to  produc¬ 
tivity.  Strive  to  work  in  an  unclassified  area  as  long  as  possible.  Also 
make  sure  that  there  are  sufficient  cleared  people  available  to  do  the 
work  and  escort  others  after  the  posture  is  upgraded. 

•  Government  procurement  programs  may  be  required.  Identify  them 
early  in  the  process.  Requirement  waivers  are  time  consuming  and 
costly. 

•  Keep  in  mind  that  procuring  or  fabricating  items  before  understanding 
the  requirements  may  make  the  project  more  expensive.  You  may  need 
to  purchase  long  lead-time  items  before  you  are  certain  you  need  them 
or  understand  the  related  costs.  This  can  lead  to  wasted  time  and 
funds  (e.g.,  returning  unusable  equipment).  Be  aware  of  items  that 
have  long-lead  ordering  time;  if  possible,  confirm  these  items. 

•  Find  out  early  if  there  are  personnel  and  site  requirements  that  must  be 
followed.  In  this  project,  the  Raytheon  facility  was  a  union  shop,  and 
therefore  had  to  be  involved  in  all  property  movement,  which  slowed 
down  work  and  frustrated  the  development  team.  Later  in  the  project, 
we  had  insufficient  cleared  personnel  for  union  escort  duty. 

•  Get  the  processes  for  configuration  management  in  place  as  early  as 
possible  or  this  will  become  a  problem  at  signoff.  This  should  include 
documentation,  license  management,  and  property  control. 

•  Some  enabling  resources  may  not  be  project  deliverables,  e.g.,  a  local 
file  server  for  file  sharing,  printers,  and  large  format  plotters.  Get 
needed  resources  as  soon  as  possible  to  leverage  their  potential  as  long 
as  possible.  Check  existing  resources  to  see  if  these  items  are  available. 
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•  Processes  that  are  not  critical  early  on  but  that  will  be  critical  later 
should  be  identified  early  and  monitored  often  to  avoid  surprises  and 
setbacks. 

•  Outsource  rather  than  build  in-house  if  it  makes  sense.  We  built  CAT-5 
cables  in  house,  and  then  had  to  order  commercial  off-the-shelf  cables 
because  the  self-built  cables  had  quality  control  problems.  In  this  case, 
building  the  cables  in-house  wasted  labor,  time,  and  money. 

•  Recognize  tasks  where  parallel  efforts  can  apply.  However,  keep  in 
mind  that  certain  processes  or  sets  of  processes  are  inherently  sequen¬ 
tial.  If  one  engineer  could  implement  a  product  in  10  months,  giving 
the  process  to  10  engineers  does  not  necessarily  mean  that  the  job  will 
be  done  in  1  month. 

•  Testing  and  evaluation  should  be  designed  in  from  the  beginning  and 
should  start  at  the  beginning  of  the  build  process.  If  "new  technology" 
is  being  used,  make  sure  to  test  it  in  the  target  environment  before  com¬ 
mitting  to  the  technology.  Legacy  systems  do  not  always  work  as 
intended  when  on  newer  platforms  or  operating  systems. 

•  Designing  early  for  the  test  and  evaluation  phase  helps  in  the  long  run 
for  customer  acceptance.  Providing  the  customer  with  a  strong  set  of 
test  documentation  will  help  with  the  acceptance  signoff;  this  can  also 
affect  safety  and  security. 

•  Integrated  Logistics  Support  (ILS)  cannot  be  ignored  or  drastically 
reduced.  Issues  of  safety,  spares,  documentation,  and  training  should 
be  built  in  and  kept  updated  throughout  development.  Safety  issues 
must  be  followed  and  fixed  immediately.  A  "zero  spares"  policy  can 
slow  the  process  by  causing  delays  when  equipment  fails. 

Interaction  with  the  Customer 

•  Always  consider  customer  needs.  For  example,  the  customer  had  just 
removed  cameras  from  their  JWICS  J2  network  at  CENTCOM.  We 
"forced"  a  camera  on  them  at  the  user  level  because  of  a  higher  com¬ 
mand  level  requirement  for  desktop  collaboration.  This  required  a  sig¬ 
nificant  effort  to  produce  a  client  base  load  that  could  use  the  camera 
even  though  the  camera  would  not  work  with  their  applications.  We 
expected  their  applications  would  run  in  a  VMWare  session.  This  con¬ 
fused  the  users,  and  the  applications  ran  considerably  slower.  We 
ended  up  removing  the  cameras  and  reverting  to  their  approved  base¬ 
line,  running  on  NT  not  a  VMWare  NT  session. 

•  Do  not  change  applications  unnecessarily.  For  example,  the  customer 
was  using  Norton  Ghost™  for  creating  images  of  their  machines.  We 
chose  PowerQuest  DeployCenter™  instead.  Although  this  worked, 
there  was  not  an  overriding  reason  to  use  one  over  the  other.  Using 
DeployCenter  caused  the  user  to  have  to  learn  a  new  product,  and 
some  incompatibility  issues  were  discovered  along  the  way. 

Testing  and  Verification 

•  Make  sure  components  are  tested  for  actual  requirements.  For  example, 
network  cables  built  by  the  team  were  tested  for  100-MHz  operation 
but  were  required  to  run  at  1-GHz  operation.  This  interacted  with  the 
new  communications  and  applications  components,  and  software 
reduced  confidence  at  integration  and  troubleshooting  time,  causing 
the  team  to  rewire  multiple  times.  This  costs  time  and  money. 

•  Create  test  plans  (and  training  materials)  before  and  during  implemen¬ 
tation.  Identify  interdependencies  (software  and  hardware)  so  regres¬ 
sion  testing  can  be  minimized. 
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•  Some  testing  (experimentation)  may  need  to  be  done  early.  If  you  are 
going  to  be  using  "new  technology,"  make  sure  to  test  it  in  the  target 
environment  before  committing  to  it. 


CONCLUSION 

This  paper  has  explored  some  of  the  issues  and  insights  to  large  team 
design  and  implementation  of  a  complex  product  under  unrealistic  time 
constraints  while  maintaining  levels  of  best  practice  engineering.  While 
we  attempted  to  prioritize,  balance,  and  properly  software  engineer 
CDHQ,  there  were  many  wrong  turns  made  due  to  the  constraints  and 
complexity  of  the  product  and  team.  It  is  hoped  this  will  provide  insight  to 
other  scientists  and  engineers  so  that  they  may  learn  from  our  experience. 
While  not  the  perfect  engineering  delivery,  the  dedication  and  patriotic 
nature  of  those  involved  allowed  it  to  be  carried  to  completion,  where  it 
continues  to  serve  the  warfighter  today. 
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THE  KNOWLEDGE-ACQUISITION  BOTTLENECK 

For  artificial  intelligence  to  become  useful  in  practical  applications  and 
environments,  it  is  necessary  to  identify,  document,  and  integrate  into 
automated  systems  the  knowledge  that  people  use  to  solve  problems. 

This  process,  called  knowledge  acquisition,  has  become  a  bottleneck  in 
the  development  of  artificial  intelligence-based  systems  because  knowl¬ 
edge  acquisition  is  difficult,  labor  intensive,  and  error  prone.  This  paper 
proposes  a  solution  to  the  knowledge-acquisition  bottleneck.  It  describes 
research  in  the  development  of  a  general  methodology  for  modeling  and 
representing  an  expert’s  problem-solving  process  in  a  knowledge-based 
agent  through  teaching-based  ontology  formation  and  rule  learning. 
Based  on  the  task-reduction  paradigm  of  problem  solving,  this  methodol¬ 
ogy  is  designed  to  accomplish  the  following  functions: 

•  Helps  the  domain  expert  conceptualize  the  problem-solving  process. 

•  Allows  experts  to  document,  order,  and  justify  their  decision-making 
process. 

•  Facilitates  directly  the  expression  of  this  information  to  the  agent. 

•  Governs  the  entire  agent-training  and  knowledge-base  development 
process. 

•  Facilitates  natural  interactions  for  the  agent  to  learn  complex  problem¬ 
solving  rules. 

•  Supports  reuse  of  ontologies  and  extension  of  ontologies  for  the 
problem  domain. 

EXPERT  DECISION  MAKING  AND  TASK  REDUCTION 

Experts  constantly  need  to  make  complex  decisions  rapidly.  Despite  the 
complexity  of  the  problem  and  the  variety  of  approaches  available,  one 
methodology  that  is  seen  consistently  in  explaining  and  documenting  the 
accessible  parts  of  the  decision-making  processes  is  task  reduction.  Kirlik 
et  al.  have  suggested  that  task-simplification  strategies  based  mainly  on 
perception  and  pattern  recognition  are  fundamental  to  the  novice-expert 
shift  in  dynamic  decision  making  [1].  Task  reduction  is  the  process  of 
taking  a  complex  problem  and  reducing  it  to  a  series  of  less  and  less  com¬ 
plex  problems  until  relatively  simple  problems  remain  for  which  we  have 
enough  information  to  reach  a  conclusion  [2, 3].  One  key  challenge  was 
the  collection  and  representation  of  this  type  of  decision-making  infor¬ 
mation.  In  a  wide  variety  of  domains,  experts  face  cognitively  demanding 


ABSTRACT 

This  paper  describes  Disciple, 
a  decision  aid  based  on  artifi¬ 
cial  intelligence  techniques, 
that  subject-matter  experts  can 
train  and  use  when  making 
decisions  under  stressful,  com¬ 
plex,  and  constrained  condi¬ 
tions.  The  tool  was  developed 
and  used  under  the  Defense 
Advanced  Research  Projects 
Agency’s  High  Performance 
Knowledge  Base  and  Rapid 
Knowledge  Formation  pro¬ 
grams  at  the  George  Mason 
University  Learning  Agents 
Laboratory.  Disciple  could 
contribute  to  enhanced  deci¬ 
sion-making  efficiency  as  a 
decision  aid  in  various 
domains,  including  military 
battle  planning,  as  demonstrat¬ 
ed  by  experiments  described  in 
this  paper.  The  paper  con¬ 
cludes  with  a  discussion  of 
future  research  in  decision- 
support  application  tools. 
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tasks  that  have  costly  consequences  for  poor  or  ineffective  performance 
[4].  Table  1  lists  examples  of  challenging  tasks  and  task-reduction  tech¬ 
niques  that  experts  use  to  approach  and  solve  problems.  Any  automated 
decision-support  tool  needs  to  accommodate  various  task-reduction  tech¬ 
niques  of  the  application  domain. 


TABLE  1 .  Examples  of  decision-making  under  pressure  and  task  reduction. 


1 

Attribute 

Military  Domain 

Medical  Domain 

1 

Manufacturing  Domain 

Knowledge  Acquisition 

Intelligence  gathering 

Diagnostic  tests,  examinations 

Situation  assessment 

Source  of  Decision 
Pressure 

Public  perception,  expectation 
of  commanding  officers, 
uncertainty  of  battle  (the 
"fog  of  war") 

Patient  expectation,  clinical 
schedule,  progress  of  disease 

Customer  expectation,  production 
schedule,  marketplace  competition, 
environmental  dangers,  government 
regulations 

Consequence 
of  Error 

Failed  military  mission,  loss 
of  assets,  wartime  casualties, 
multiple  fatalities 

Untreated  disease,  treating 
the  wrong  disease,  selecting 
wrong  treatment  cost, 
suffering,  fatalities 

Damage  to  products  or  materials, 
cancelled  contracts,  injury  accidents, 
litigation,  financial  damage,  fatalities 

Examples 

Developing  a  course  of  action 
(COA)  or  plan  for  a  military 
operation 

Diagnosing  a  disease  when 
multiple  diseases  are  present 
or  when  a  group  of  symptoms 
has  a  combination  of  causes 

Determining  a  strategy  to  transport 
heavy,  expensive,  unstable,  hazardous 
and/or  fragile  materials  or  products 
in  a  factory/ shipyard 

Task-reduction 

Techniques 

COA  sequence  according  to 
published  doctrine  [5] 

Step-by-step  medical 
procedure 

Safety  checklists  and 
engineering  guidelines 

DARPA  HPKB  AND  DISCIPLE 

The  Defense  Advanced  Research  Projects  Agency  (DARPA)  High 
Performance  Knowledge  Base  (HPKB)  program  ran  from  1997  to  1999 
[5].  The  goal  of  HPKB  was  to  produce  the  technology  needed  for  the 
rapid  construction  of  large  knowledge  bases  (with  many  thousands  of 
axioms)  that  provide  comprehensive  coverage  of  topics  of  interest,  are 
reusable  by  multiple  applications  with  diverse  problem-solving  strategies, 
and  are  maintainable  in  rapidly  changing  environments.  HPKB  partici¬ 
pants  were  given  the  challenge  of  solving  a  selection  of  knowledge-based 
problems  in  a  particular  domain  and  then  modifying  their  systems  quick¬ 
ly  to  solve  further  problems  in  the  same  domain.  (See,  for  example,  [5] 
and  [6]).  The  challenge  problems  tested  the  speed  and  efficiency  with 
which  large  knowledge  bases  could  be  built. 

The  George  Mason  University  (GMU)  Learning  Agents  Laboratory’s 
(LALAB)  approach  to  HPKB  was  based  on  the  Disciple  Toolkit. 
Disciple’s  history,  capabilities,  inner  workings,  future  research  directions, 
and  publications  are  described  in  detail  in  [7]  and  [3].  More  information 
can  be  found  on  the  GMU  LALAB  web  page,  http://lalab.gmu.edu. 
Disciple  is  a  theory,  methodology,  and  agent  shell  for  rapid  development 
of  knowledge  bases  and  knowledge-based  agents  by  domain  experts  with 
limited  assistance  from  knowledge  engineers.  The  Disciple  agent  shell 
consists  of  an  integrated  set  of  knowledge  acquisition,  learning,  and 
problem-solving  modules  for  a  generic  knowledge  base  structured  into 
two  main  components:  an  object  ontology  that  defines  the  concepts  from 
a  specific  application  domain  and  a  set  of  problem-solving  rules  expressed 
with  these  concepts.  The  process  of  developing  a  specific  Disciple  agent, 
starting  from  the  Disciple  shell,  relies  on  importing  ontological  knowledge 
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from  existing  knowledge  repositories,  and  on  teaching  the  agent  how  to 
perform  various  tasks.  This  paradigm  allows  an  expert  to  teach  the  agent 
as  though  it  were  a  human  apprentice  by  giving  the  agent  specific  exam¬ 
ples  of  tasks  and  solutions,  by  providing  explanations  of  these  solutions, 
and  by  supervising  the  agent  as  it  performs  new  tasks.  In  HPKB  applica¬ 
tions,  a  military  expert  taught  Disciple  to  perform  various  tasks  in  a  way 
that  resembles  how  the  expert  would  teach  a  novice.  Disciple  learns  from 
the  expert,  building  and  improving  its  knowledge  base  and  expanding  its 
problem-solving  capability.  To  conduct  productive,  interactive  training 
episodes  with  Disciple,  the  experts  must  understand  and  document  their 
decision-making  process  with  respect  to  the  examples  to  be  used  in  the 
training  episode. 

HPKB  COA  CRITIQUING 

The  problem  domain  for  one  of  the  HPKB  challenge  problems  was  the 
critiquing  of  military  courses  of  action  (COA).  A  military  COA  is  a  pre¬ 
liminary  outline  of  a  plan  for  how  a  military  unit  might  attempt  to 
accomplish  a  mission.  The  example  COAs  used  for  this  research  and  pro¬ 
vided  by  the  U.S.  Army  were  specified  in  standard  military  formats  con¬ 
sisting  of  a  multi-paragraph  textual  description  of  the  COA  and  a  graphi¬ 
cal  representation  of  the  COA  in  the  form  of  a  sketch  using  standardized 
symbols  for  units,  activities,  and  geo-spatial  relationships.  The  developed 
Disciple  critiquer  identifies  strengths  and  weaknesses  of  a  COA  with 
respect  to  the  principles  of  war  and  the  tenets  of  army  operations  [8]. 

(See,  for  example,  [5]).  According  to  U.S.  Army  doctrine,  the  nine  princi¬ 
ples  of  war  are  objective,  offensive,  mass,  economy  of  force,  maneuver, 
unity  of  command,  security,  surprise,  and  simplicity.  The  Disciple-COA 
critiquing  agent  was  developed  to  act  as  an  assistant  to  a  military  com¬ 
mander  and  staff,  helping  them  to  choose  the  best  COA  for  a  particular 
mission.  Application  of  the  principles  of  war  and  the  tenets  of  army 
operations  as  described  in  [8]  is  just  the  beginning  of  a  good  critique  of  a 
COA  or  plan.  GMU’s  goal  was  to  create  a  Disciple  agent  that  contained 
the  common  understanding  of  the  principles  and  tenets  while  retaining 
sufficient  flexibility  to  allow  rapid  personalization  by  the  experts  training 
and  using  the  agent. 


AGENT  DEVELOPMENT  METHODOLOGY 

The  importance  and  usefulness  of  this  methodology  for  modeling  and 
representing  an  expert  decision-making  process,  (as  expressed  in  the  thor¬ 
ough  and  accurate  task-reduction  steps,  questions,  and  answers  that  the 
expert  provides),  is  captured  in  our  high-level  research  goals.  Our  objec¬ 
tive  is  to  have  a  domain  expert  interact  directly  and  independently  with 
the  agent-building  shell  to  train  an  agent  to  solve  complex  problems. 
Experts  type  natural  language  text,  use  mouse  clicks  to  provide  hints  for 
explanation  generation,  and  use  mouse  clicks  to  identify  and  select  cor¬ 
rect  explanations.  We  do  not  expect  an  expert  to  create  formal  sentences 
for  explanations  or  explicitly  create  rules  in  machine-executable  language. 
This  modeling  provides  the  basis  for  the  expert-agent  interaction.  For  a 
detailed  explanation  of  the  following  domain-modeling  and  representa¬ 
tion-methodology  steps,  see  [3]. 

1.  Identify  the  high-level  problem  to  be  solved. 

2.  Identify  categories  of  potential  solutions. 
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3.  Identify  a  specific  example  problem  to  be  solved. 

4.  Brainstorm  potential  solutions  for  the  example  problem  within  a 
category  of  solution. 

5.  Select  a  potential  solution  to  be  modeled. 

6.  Identify  the  complete  set  of  task-reduction  steps  for  that  potential 
solution. 

7.  Identify  a  question  and  answer  that  justifies  progression  from  one 
step  to  another  in  the  task-reduction  solution  path. 

8.  Identify  concepts  and  features  for  Disciple’s  ontology. 

9.  Use  the  questions  and  answers  as  the  basis  for  hints  provided  to 
Disciple  (i.e.,  selecting  relevant  concepts,  instances,  and  relation¬ 
ships). 

10.  Use  the  questions  and  answers  to  identify  correct  justifications 
among  the  justifications  provided  by  Disciple  during  rule  develop¬ 
ment  or  refinement. 

11.  Repeat  the  process  for  other  solution  paths. 

12.  Check  solutions  and  refine  rules  for  other  data  sets. 

EXPERIMENTAL  TRIALS  AND  RESULTS 

The  HPKB  evaluation  results  are  documented  in  [9]  and  [3].  In  summary, 
these  results  show  that  Disciple-based  agents  built  by  teams  of  subject- 
matter  experts  (SMEs)  and  knowledge  engineers  using  early  versions  of 
the  methodology  were  highly  effective  in  solving  complex  problems  and 
produced  very  high  knowledge-acquisition  rates.  A  knowledge-acquisition 
experiment  at  the  U.S.  Army  Battle  Command  Battle  Laboratory  at  Fort 
Leavenworth,  KS,  determined  the  extent  to  which  SMEs  with  no  knowl¬ 
edge-engineering  experience  could  train  Disciple  to  critique  a  COA. 
SMEs  modified  and  used  models  prepared  by  other  military  experts  for 
our  EIPKB  evaluations,  and  tested  whether  these  models  were  appropri¬ 
ate  and  sufficient  to  support  SME  attempts  to  develop  and  train  Disciple- 
based  agents.  The  SMEs  were  U.S.  Army  combat  arms  officers  with  16  to 
22  years  of  military  service.  The  experts  received  briefings  that  explained 
artificial  intelligence,  research  goals,  experimental  design,  Disciple,  and 
the  task- reduction  models  for  problem  solving.  (See,  for  example,  [10] 
and  [11]). 

During  the  experiment,  the  military  SMEs  each  separately  taught  Disciple 
to  critique  a  COA  with  respect  to  the  principles  of  offensive  and  security. 
Starting  with  a  conceptual  modeling  of  the  critiquing  process  for  these 
two  principles,  they  each  independently  developed  a  knowledge  base  in 
a  single  day.  For  one  expert,  training  for  the  principle  of  offensive  con¬ 
sisted  of  101  minutes  of  expert-Disciple  interactions.  Disciple  learned 
14  tasks  and  14  rules.  Training  for  security  consisted  of  72  minutes  of 
expert-Disciple  interactions.  Disciple  learned  14  tasks  and  12  rules.  The 
knowledge  engineer  provided  very  limited  training  assistance.  The 
knowledge-acquisition  rates  obtained  during  the  experiment  were  very 
high  (~9  tasks  and  8  rules/hour  expert).  This  knowledge-acquistion 
experiment  is  one  of  the  most  significant  accomplishments  of  our 
research.  To  our  knowledge,  it  is  the  first  time  a  SME  with  no  prior 
knowledge  engineering  experience  has  succeeded  in  extending  a  signifi¬ 
cant  knowledge  base  in  a  very  short  period  of  time,  without  significant 
support  from  a  knowledge  engineer. 
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CONCLUSION 

Our  experimental  results  show  that  we  have  developed  a  general- 
purpose  methodology  and  tool  for  expert  knowledge  acquisition  based 
on  apprenticeship  multi-strategy  learning  in  a  mixed-initiative  frame¬ 
work.  This  methodology  enhances  the  ability  of  a  domain  expert  with 
very  little  knowledge  engineering  experience  to  build  a  knowledge  base 
efficiently.  The  Disciple  methodology  accomplishes  six  major  functions: 

(1)  helps  the  domain  expert  conceptualize  the  problem-solving  process; 

(2)  allows  an  expert  to  document,  order,  and  justify  their  decision-making 
process;  (3)  facilitates  directly  the  expression  of  this  information  to  the 
agent;  (4)  governs  the  entire  agent  training  and  knowledge  base  develop¬ 
ment  process;  (5)  facilitates  natural  interactions  that  allow  the  agent  to 
learn  complex  problem-solving  rules,  and  extend  and  correct  the  domain 
knowledge  base;  and  (6)  supports  the  reuse  of  existing  ontologies  and  the 
extension  of  these  ontologies  for  the  problem  domain. 

DIRECTIONS  FOR  FUTURE  RESEARCH 

Modeling  and  ontology-acquisition  activities  can  be  automated,  particu¬ 
larly  with  regard  to  the  direct  capture  of  ontology  elements  consisting  of 
concepts,  objects,  and  features.  Disciple  can  be  modified  to  learn  more 
complex  rules  with  improved  methods.  Disciple  is  being  extended  in  an 
attempt  to  create  intelligent  agents  to  help  identify  candidate  strategic 
centers  of  gravity  for  historic  and  modern  crisis  and  wartime  scenarios. 
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INTRODUCTION 

This  paper  provides  a  broad  overview  of  collaborative  systems  and  the 
human  collaborative  process  by  reviewing  the  various  research,  develop¬ 
ment,  test,  and  evaluation  activities  conducted  at  SSC  San  Diego.  These 
activities  were  undertaken  for  a  variety  of  reasons,  not  the  least  of  which 
is  the  continuing  evolution  to  a  fully  netted  force  in  which  distributed 
operational  personnel  have  the  ability  to  execute  their  collaborative  mis¬ 
sion  supported  by  the  right  information  at  the  right  time.  In  the  mid- 
1980s,  it  became  apparent  that  the  transformation  from  large-scale  VAX 
systems  to  integrated  personal  computers  (via  the  ground-breaking  work 
of  Dr.  Doug  Englebart,  Bootstrap  Institute)  would  drive  the  way  our 
forces  would  define  the  operational  situation,  analyze  its  potential,  develop 
courses  of  action  relevant  to  various  scenarios,  and  craft  multi-disciplinary 
joint  execution.  The  ensuing  goal  was  the  electronic  "netting"  of  the  dis¬ 
parate  elements  of  a  mission,  so  that  a  seamless  integration  of  computing 
resources,  situation  assessments  (consistent  operational  pictures),  per¬ 
ceived  value  of  actions,  and  the  actions  themselves  would  combine  syner- 
gistically.  To  achieve  this  seamless  integration,  two  things  must  happen  at 
the  warfighter  level.  Distributed  warfighters  must  be  able  to  develop  a 
consistent  situation  assessment  (in  other  words,  agree  on  the  relative 
meaning  of  the  tactical  picture),  and  they  must  develop  a  coordinated,  or 
more  precisely,  collaborative,  process  in  which  their  actions  can  be  syn¬ 
chronized  to  achieve  mission  objectives.  This  paper  assumes  that  the 
development  of  distributed,  collaborative  processes  is  instrumental  to  the 
development  of  a  shared  understanding  of  the  situation  and  that  the  exe¬ 
cution  of  coordinated  activities  is  essential  to  the  successful  achievement 
of  the  joint  mission.  These  collaborative  processes  have  many  definitions, 
but  the  one  we  prefer  was  coined  by  a  warfighter  himself  and  captures 
the  essence  of  that  process: 

"  Collaboration  entails  the  activities  of  two  or  more  people  working  on  a 
common  goal/objective/object,  with  shared  data,  in  which  a  product  is 
left  behind  at  the  end  of  the  process.  In  this  context,  the  object  may  be  a 
decision,  a  document,  an  image,  a  shared  understanding  of  the  situation 
(not  necessarily,  an  agreement),  or  a  plan  of  action."1  [1], 

Add  to  this  definition  the  prospect  of  distributing  team  members  (virtual 
teams),  and  you  require  an  engineering  component  (networked  systems) 
to  the  human  process.  In  other  words,  the  team  relies  on  the  network  for 


ABSTRACT 

SSC  San  Diego  conducted  one 
of  its  first  collaborative  tech¬ 
nology  demonstrations  during 
Secure  Tactical  Data 
Networks- 1994  (STDN-4), 
demonstrating  the  transport  of 
video  images  from  shore  to 
ship  at  3-12  frames  per 
minute.  Collaborative  systems 
research  has  grown  steadily 
throughout  the  Center  during 
the  ensuing  years.  Currently, 
several  complex  collaborative 
technologies  are  being  demon¬ 
strated,  including  ship-to-ship 
collaboration  in  the  current 
Joint  Warrior  Interoperability 
Demonstration  2003.  With  the 
proliferation  of  commercial 
collaboration  technologies,  the 
growing  impact  of  joint  opera¬ 
tions  on  Navy  missions,  and 
the  need  to  constantly  refresh 
and  integrate  these  technolo¬ 
gies,  there  is  a  need  to  under¬ 
stand  the  history  of  research, 
development,  test,  and  evalua¬ 
tion  endeavors  in  this  area. 
Prepared  under  the  auspices  of 
the  Center  for  Command 
Technology  Transformation,  a 
cross-departmental  team  that 
monitors  and  supports  collabo¬ 
rative  technologies,  this  paper 
describes  a  decade  of  these 
activities  conducted  at  SSC 
San  Diego  (1993-2003). 


1  This  definition  was  coined  by  Jens  Jensen,  USPACOM  J3  Deputy  Director  of 
the  Operations  Planning  Team. 
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distributed  communications  and  computing  in  support  of  team  interac¬ 
tion.  Unlike  conventional  co-located  teams,  a  virtual  team  works  across 
space,  time,  and  organizational  boundaries  with  links  strengthened  by 
"webs"  of  networked  communications  technologies.  In  essence,  for 
virtual  teams,  the  network  IS  the  computer™  2  This  implies  that  the 
study  of  collaborative  processes  and  systems  is  not  only  a  study  in  dis¬ 
tributed  human  behavior,  but  also  the  study  of  the  network  architecture 
that  underlies  the  communications  and  computing  resources  used  by  the 
distributed  warfighters. 


COLLABORATION  "TECHNOLOGIES"  OVERVIEW 

Given  the  definition  above,  the  goal  of  development  activities  in  comput¬ 
erized  support  for  collaborative  team  processes  is  in  its  ability  to  support 
three  levels  of  warfighter  activity:  individualized  and  uncoordinated 
effort;  multiple  individual,  coordinated,  yet  independent  efforts;3  and 
finally,  the  truly  concerted  or  collaborative  activities.  A  musical  analogy 
to  these  three  levels  might  be  a  soloist,  a  duet  by  concert  members 
engaged  in  a  scripted  opera,  and  a  jazz  ensemble  engaged  in  a  continuing 
renegotiation  of  the  musical  product  in  real  time. 

Capabilities  (and  attendant  technologies)  that  have  evolved  to  support 
collaborative  processes  involve  nine  general  categories.  Several  have 
appeared  elsewhere  [2,  3],  but  the  list  below  has  emerged  through  our 
own  research  into  the  field. 

Information  Sharing:  This  category  includes  tools  that  provide  a  common 
information  space,  sometimes  with  tailored  workgroup  applications,  such 
as  Lotus  Notes®  or  non-interactive  web  pages. 

Electronic  Conferencing:  Audio-video  (real-time)  conferencing  and  dis¬ 
cussion  databases  (chat  rooms)  dominate  this  category,  although  the  lines 
are  blurring  with  the  advent  of  instant  messaging,  as  well  as  speech  to  text 
and  text  translation  technologies. 

Data  Conferencing:  The  least  understood,  but  perhaps  one  of  the  oldest 
technologies,  shared  whiteboarding,  may  become  one  of  the  most  power¬ 
ful  tools  to  be  developed.  The  ability  of  an  application  to  provide  a 
working  surface  that  several  users  can  control  at  the  same  time  is 
unprecedented  in  the  development  of  collaborative  experiences.  Similarly, 
the  dynamics  of  application  sharing  and  interacting  with  data  in  real  time 
has  only  just  begun  to  be  investigated.  The  initial  attempt  to  provide  a 
WYSIWIS  (what  you  see  is  what  I  see)  surface  has  been  the  predominant 
theme  in  situation  assessment  technologies. 

Group  Authoring:  Shared  text  documents  for  group  authoring  and  edit¬ 
ing  have  been  the  domain  of  intelligence  analysis,  doctrine  and  policy 
development,  and  document-dominant  activities,  such  as  those  using 
collaborative  Excel®  spreadsheets.  This  has  been  the  predominant  appli¬ 
cation  in  business  and  staff  planning  environments. 


2  Borrowing  the  tag  line  from  Sun  Microsystems,  this  concept  is  made  apparent 
when  one  thinks  of  the  isolated  application  test:  installling  an  application  and 
testing  usability  on  a  single  workstation.  With  collaborative  applications,  you 
must  be  "networked"  or  you  have  little  collaboration.  This  applies  equally  to 
telephones,  fax  machines,  and  the  carrier  pigeon. 

3  We  have  taken  the  liberty  of  defining  coordination  as  the  exchange  of  informa¬ 
tion  that  does  not  entail  an  elaborated  exchange,  which  generally  is  the  intuitive 
hallmark  of  collaboration.  For  example,  target  location  data  reported  from  a 
forward  element  to  a  rear  echelon  would  be  a  coordination  act.  It  becomes 
collaborative  if  there  is  an  extended  exchange  during  which  clarification  and 
possible  disagreement  and  search  for  common  understanding  ensue. 


42 


COMMAND  AND  CONTROL 


Group  Calendaring  and  Scheduling:  Sharing,  viewing,  and  editing  team- 
members’  schedules/calendars  has  been  a  mainstay  of  office  systems  for 
some  time.  Typically,  two  issues  arise:  at  a  personal  level,  people  "lie" 
about  their  schedules  in  order  to  protect  their  time  and,  at  a  broader 
level,  the  ability  to  transfer  dynamic  changes  to  schedule  information 
across  a  variety  of  peripherals  (e.g.,  personal  digital  assistants)  has  not 
been  a  smooth  path.  Exercise  planners  were  among  the  first  to  exploit 
this  capability,  with  a  rapid  rise  in  maintaining  battle  rhythm  manage¬ 
ment  to  follow. 

Workflow:  Automating  routine  tasks  with  user  notifications  and  alerts  is 
a  growing  area  (having  evolved  from  the  manufacturing  sector)  and  may 
be  well  suited  to  support  some  tactical  processes  [4].4  Research  and 
development  is  increasing  in  this  area,  given  the  current  interest  in  "  bots " 
and  artificial  intelligence  software  agents  [5]. 

Group  Decision  Systems:  Focusing  on  group  process  support  (e.g.,  brain¬ 
storming,  organizing  groups,  consolidating  information  or  supporting  a 
group  decision)  has  been  the  domain  of  group  decision  support  systems. 
The  area  saw  an  exponential  increase  in  research  and  development  in  the 
late  1980s  through  mid  1990s,  but  has  lain  dormant  for  the  past  3  to  5 
years  [6,  7]. 

Virtual  Environments:  The  focus  of  development  efforts  in  the  collabora¬ 
tive  systems  arena  has  been  on  collections  of  group  support  functions  in 
one  application.  There  were  only  four  predominant  systems  in  1997  5  but 
the  area  in  general  has  now  evolved  into  many  separate  multi-function 
applications  specific  to  divergent  work  areas  (e.g.,  finance,  insurance, 
medical,  etc.).  However,  five  virtual  environments  dominate  the  commer¬ 
cial  market,  as  well  as  the  military  area. 

Simulated  Environments:  These  are  virtual  environments  with  a  heavy 
emphasis  on  three-dimensional  (3-D)  imaging  and  have  a  strong  emphasis 
on  the  reality  of  the  3-D  views.  The  modeling  and  simulation  community 
predominates  this  area  in  their  quest  for  the  perfect  immersive  training 
system. 


RESEARCH  IN  COLLABORATIVE  PROCESSES 

A  short  synopsis  of  a  variety  of  research  activities  conducted  at  SSC  San 
Diego  in  varying  areas  of  distributed  collaborative  processes  include  the 
following  projects  (funding  sources  are  given  in  parentheses): 

1993-1995:  Distributed  Situation  Assessment  among  Distributed  Team 
Experts  (Office  of  Naval  Research  [ONR]).  This  was  a  study  of  the  use 
of  electronic  whiteboards  and  its  application  to  collaborative  situation 
assessment  activities,  given  uniquely  held  information  by  individuals  in  a 
team  of  functional  experts  [8]  distributed  across  an  amphibious  readiness 
group  within  a  battle  group. 


4  The  research  headed  by  Dr.  Glenn  Osga  on  the  Land  Attack  Combat  System 
Human-Computer  Interface  project  is  a  case  in  point. 

5  By  1997,  the  most  comprehansive  collaborative  virtual  environments  associated 
with  the  military  were  LambdaMOO  by  the  Xerox  Palo  Alto  Research  Center 
(PARC);  Collaborative  Virtual  Workspace™  by  MITRE;  wOrlds  (later  called 
OrbitGold)  by  the  University  of  Illinois,  Urbana-Champaign  (later  transferred 
to  the  University  of  Queensland);  Odyssey  Collaboration  System,  built  at  SSC 
San  Diego;  and  E-Room  (a  now  defunct  commercial  vendor).  All  owe  their 
evolution  to  the  original  text-based  multi-user  domains/MUD-Object  Oriented 
(MUDS/MOOS)  used  on  college  campuses  during  the  early  1980s  before  some 
of  them  evolved  into  very  popular  online,  multi-user  Internet  games. 
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1996-1998:  Distributed  Command  and  Control  Collaboration  Joint  Task 
Force  Advanced  Technology  Demonstration  (Defense  Advanced  Projects 
Agency  [DARPA]).  This  demonstration  project  identified  current  collab¬ 
orative  technologies,  leveraged  the  philosophy  and  insight  of  Dr.  Douglas 
Englebart,  and  analyzed  the  ability  of  collaborative  systems  to  support 
distributed  command  and  control  planning  processes.  This  was  one  of  the 
first  integrated  system  development  efforts  to  support  distributed  collab¬ 
orative  planning. 

1999-2001:  Decision  Support  System  for  Coalition  Operations  (ONR). 
This  was  a  research  analysis,  as  well  as  a  development  effort,  focused  on 
supporting  coalition  planners  having  to  deal  with  cultural  differences 
during  distributed  team  activities  while  engaged  in  operations  other  than 
war  (e.g.,  humanitarian  assistance  and  disaster  relief.) 

2001-2003:  Commercial  Off-the-Shelf  (COTS)  Collaboration  Tools  for 
CL  Then  and  Likely  Future.  This  is  an  ongoing  effort  culminating  in  the 
tracking  of  over  600  collaboration  and  knowledge  management  web 
sites.6 

2001- 2003:  Decision  Making  Constructs  in  a  Distributed  Environment 
(ONR).  A  study  of  the  impact  of  (un)shared  information  on  the  quality 
of  command  and  control  decision  making  and  how  uniquely  held  infor¬ 
mation  can  be  shared  and  integrated  into  the  collective  knowledge  of  a 
distributed  command  and  control  decision-making  team. 

1996-2000:  Command  and  Control  Multi-User  Virtual  Environments 
(ONR).  The  research  and  development  of  a  multi-user  virtual  environ¬ 
ment  applicable  to  U.S.  Pacific  Command  J3  (USPACOM  J3)  Operations 
Planning  Team  distributed  activities,  progressing  from  "flat"  web  services 
to  3-D  client-server  architectures,  culminating  in  a  federated  server  archi¬ 
tecture,  providing  user-constructed  virtual  command  centers,  staff  offices, 
and  collaborative  web  services. 

1999-2001:  Distributed  Interactive  Environments:  Heterogeneous  sys¬ 
tems  Research  and  Development  (ONR).  This  research  focused  on  the 
evolution  of  multi-user  virtual  environments  to  encompass  massively 
multiple-player  Internet  gaming  network  architectures,  as  well  as  a  gam¬ 
ing  infrastructure  in  support  of  online  distributed  rehearsals  for  the 
USPACOM  J3  Operations  Planning  Team.  The  game  was  based  on  Dr. 
Richard  Duke’s  gaming  language  and  past  experience  building  an  organi¬ 
zational  game  for  the  Joint  Chiefs  of  Staff.  [9] 

2002- 2003:  Virtual  and  Physical  Command  Centers  (ONR).  The  prolif¬ 
eration  of  mobile  and  temporary  command  centers,  alongside  the  rise  of 
virtual  (non-geolocated)  command  centers,  has  led  to  our  research  into 
their  implications  for  battle  rhythm  management  among  distributed  joint 
forces.  Furthermore,  we  are  analyzing  the  applicability  of  peer-to-peer 
computing  architectures  [10]  to  this  new  command  environment. 

DEVELOPMENT  IN  COLLABORATIVE  SYSTEMS 

Alongside  the  varying  research  efforts  have  been  several  collaboration 
system  development  efforts.  These  are  a  sample  of  those  undertaken 
from  1993  through  2003. 

Theater  Area  Replanning  Graphical  Execution  Toolkit  (TARGET). 
TARGET  was  one  of  the  first  applications  of  a  Unix-based  video- 
audio-whiteboard  "desktop"  conferencing  system  for  shipboard  use. 


6  Dr.  George  Seymour,  SSC  San  Diego  Code  244210,  has  been  tracking  these  sites 
independently. 
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TARGET  was  demonstrated  during  the  Secure  Tactical  Data  Networks- 
1994  demonstration. 

Common  Operational  Modeling,  Planning,  and  Simulation  Strategy 
(COMPASS).  COMPASS  was  middleware  developed  to  bring  distributed 
collaborative  planning,  as  well  as  modeling  and  simulation  services  to  a 
wide  range  of  command,  control,  communications,  computers  and  intelli¬ 
gence  (C4I)  systems.  The  focus  was  on  building  a  bridge  to  provide  inter¬ 
operability  among  diverse  protocols. 

Odyssey  Collaboration  System  (OCS).  This  collaborative  virtual  envi¬ 
ronment  was  the  first  to  be  built  completely  in  Java™,  exploiting 
LambdaMOO  architecture.  It  provided  a  variety  of  collaboration  services 
and  was  built  under  the  Command  and  Control  Multi-User  Virtual 
Environments  project  for  use  by  the  USPACOM  J3  Operations  Planning 
Team.  It  continues  to  be  used  at  U.S.  Central  Command  (USCENT- 
COM)  and  has  been  employed  by  the  Centers  for  Disease  Control. 

Distributed  Computing  and  Collaboration  Framework.  This  develop¬ 
ment  effort  produced  a  software  package  of  peer-to-peer  architecture 
design  requirements  and  the  testing  of  a  multi-transport  layer  (focused 
initially  on  the  user  datagram  protocol  [UDP]  and  reliable  user  datagram 
protocol  [RUDP]),  with  the  goal  of  maintaining  bandwidth  efficiency 
under  complex  network  conditions. 

Low  Bandwidth  Enhanced  Chat  (LBEC).  This  current  development 
effort,  as  part  of  the  Virtual  and  Physical  Command  Centers’  project,  is 
focusing  on  the  unique  network  requirements  of  Navy  battle  groups, 
from  "big  decks"  to  smaller  attendant  ships.  It  exploits  a  peer-to-peer 
collaborative  system  and  is  in  the  process  of  developing  "bots"  to  manage 
network  robustness  and  chat  functions  enhancements  [9]. 

DoD  Collaboration  Tool  Suite  (DCTS)  Engineering.  This  current  devel¬ 
opment  effort  is  in  support  of  the  Defense  Information  Systems  Agency’s 
(DISA’s)  Collaboration  Management  Office,  as  it  distributes  DCTS  to  all 
major  joint  command  elements.  SSC  San  Diego  is  responsible  for  the 
continued  improvement,  integration,  and  architectural  development  of 
the  suite  of  tools  that  comprise  the  DCTS. 

TEST  AND  EVALUATION  EFFORTS  IN  COLLABORATIVE  SYSTEMS 

SSC  San  Diego  has  provided  engineering  data  relevant  to  test  and  evalua¬ 
tion  of  C4I  systems  since  its  inception.  During  the  past  decade,  this  has 
also  included  evaluation  of  collaborative  systems.  Beginning  in  1993, 
when  the  Base  Closure  and  Realignment  Commission  (BRAC)  offices  in 
the  Pentagon  and  Camp  Pendleton  requested  an  analysis  of  Lotus  Notes 
(along  with  its  server  replication  software),  to  the  present,  with  engineering 
tests  of  the  latest  peer-to-peer  systems  (such  as  Groove®),  SSC  San  Diego 
has  been  able  to  provide  realistic  evaluations  of  varying  collaborative 
technologies;  the  depth  and  breadth  of  its  engineering  laboratories  allow 
emulation  of  normal  to  extreme  warfighting  conditions,  from 
submarine,  to  ship,  to  air  systems.  SSC  San  Diego  has  supported  the 
Collaboration  at  Sea  project  (with  an  emphasis  on  shipboard  employment 
of  Lotus  Domino™  for  server  replication  and  Lotus  Sametime®  for  col¬ 
laborative  services)  that  has  provided  collaborative  services  across  the 
battlegroup  where  none  were  available  previously.  The  engineering  criteria 
that  ensure  that  collaborative  systems  will  work  shipboard  are  encom¬ 
passed  in  the  Preferred  Product  List  (PPL)  process,  which  measures  com¬ 
pliancy  to  Information  Technology  for  the  21st  Century  (IT-21)  mandates. 
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Some  of  the  systems  tested  under  this  program  include  Internet  Relay 
Chat,  Microsoft  NetMeeting®,  Infoworkspace™,  Lotus  Domino/ 
Sametime,  OCS,  and  Groove. 

Several  independent  efforts  are  evaluating  current  collaborative  services 
to  understand  performance  constraints  under  varying  shipboard  (as  well 
as  joint)  conditions.  The  Virtual  and  Physical  Command  Centers  project 
is  stress-testing  Groove,  as  has  the  Grassroots  Partnership,  a  cross-SSC 
San  Diego  team  focused  on  providing  support  for  testing  of  Groove™. 
(Their  initial  test  has  been  with  Groove  on  the  Enhanced  Position 
Location  Reporting  System.)  Two  major  projects,  Commander-in-Chief 
21st  Century  (CINC21)  and  Joint  Task  Force  Wide-Area  Relay  Network 
(JTFWARNET)  have  been  testing  the  DCTS  under  network  conditions 
experienced  by  land-  and  mobile-based  command  centers.  CINC21  has 
focused  on  human-systems  evaluation,  while  JTFWARNET  has  focused 
on  systems  performance  under  extreme  conditions.  The  immediate 
future  will  entail  an  in-house  planned  test  of  the  latest  collaborative 
features  provided  by  Microsoft  under  its  Real-Time  Conferencing 
Services  system. 

Finally,  the  Center  for  Command  Technology  Transformation,  an  inter¬ 
nal  cross-departmental  SSC  San  Diego  team,  is  consolidating  Center  test 
facilities  virtually  to  provide  a  comprehensive  collaborative  system  test 
battery  (including  human-systems  evaluation  and  system  performance 
testing)  to  determine  the  optimum  configuration  for  various  collaborative 
systems  when  employed  under  extreme  conditions  (those  experienced  by 
naval  assets).  The  Center  for  Command  Technology  Transformation 
focus  is  on  providing  the  joint  warfighter  with  engineering  data  that  will 
allow  optimum  use/configuration  of  "any"  collaborative  tool  necessary 
to  do  the  job. 
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BACKGROUND 

Osga  [1]  describes  a  revolutionary  task-managed  watchstanding  system 
based  on  human  factors  and  cognitive  science  principles.  The  Multimodal 
Watch  Station  (MMWS)  is  a  response  to  the  U.S.  Navy  requirement  that 
shipboard  command  center  crews  "...complete  time-critical  and  external¬ 
ly  paced  task  assignments  in  an  accurate  and  timely  manner. "  The 
MMWS  is  made  up  of  ergonomically  designed  hardware,  a  task-explicit 
human-computer  interface  (HCI),  and  advanced  software  infrastructure. 
The  HCI  is  goal-  and  product-oriented,  task-driven,  and  features  inter¬ 
linked  display  elements.  It  encourages  users  to  engage  in  both  naturalistic 
decision-making  and  critical  thinking  [2].  Follow-on  HCI  design  work  is 
focused  on  building  high-quality,  seamless  task  visualization  tools  in  mul¬ 
tiple  operational  domains.  The  potential  benefits  include  less  dependence 
on  voice  to  transmit  data,  and  enhanced  individual  and  team  perform¬ 
ance.  This  recent  effort  is  characterized  as  a  Mission  Centered  Design 
work  interface  (MCD). 

BUILDING  A  TASK  MANAGED  SYSTEM 

In  a  task-  and  goal-explicit  HCI  such  as  the  MCD,  the  usual  office  desk¬ 
top,  window,  and  document  metaphors  are  replaced  by  graphical,  on¬ 
screen  task  representations.  MCD  tools  support  decision-making  and 
enhanced  situational  awareness.  Features  include  at-a-glance  decision 
aids,  switchable  task  contexts,  and  seamless  access  to  legacy  systems. 

Building  a  system  with  these  attributes  involves  more  than  simply  pro¬ 
ducing  better  graphical  user  interfaces  (GUIs).  The  core  logic  is  about 
task  management,  and  each  HCI  component  has  a  data  model,  a  view, 
and  a  view  controller.  Some  components  are  mini-applications,  others  are 
simpler  display  elements  (thin  components).  Data  are  provided  by  a  lega¬ 
cy  system.  Specifying  and  building  such  a  system  requires  an  HCI  soft¬ 
ware  engineering  process.  Fundamental  process  steps  include: 

•  Performing  task  analyses  to  generate  task  requirements 

•  Developing  HCI  usability  prototypes  to  generate  HCI  requirements 

•  Modeling  task  and  workload  management 

•  Selecting  a  suitable  architecture 

•  Developing  a  system  reference  implementation 


ABSTRACT 

In  fiscal  year  2002,  the  Office 
of  Naval  Research  began 
sponsoring  task-centric 
human-computer  interface 
(HCI)  design  work  in  the  land 
attack  (LA)  domain.  The  LA 
software  development  strategy 
is  to  aggressively  exploit 
advances  in  applied  computer 
science.  The  goal  is  to  move 
from  a  very  large-scale 
integration  programming 
model  to  a  distributed,  shared 
component  model.  Doing  this 
facilitates  a  shift  in  thinking 
about  the  nature  of  the  HCI. 

In  the  new  model,  the  HCI  is 
no  longer  based  on  the  com¬ 
mercial  domain ’s  application¬ 
centric  desktop  metaphor. 
Transaction- enabled  task 
management  components  run 
in  a  business  logic  tier  that  is 
decoupled  from  the  HCI.  LA 
business  logic  continues  to 
execute  as  before,  but  without 
writing  to  privately  owned 
graphical  user  interfaces.  The 
LA  functionality  and  data 
sources  are  exposed  to  the  task 
management  HCI  through 
Web  services. 


47 


Task-Managed  Watchstanding 


REQUIREMENTS 

Operational  domain  factors  that  generate  requirements  include  the 
following: 

•  Inter-  and  intra-watchstanding  team  collaboration  generates  require¬ 
ments  to  support  workgroup  activities.  Workgroup  requirements 
influence  architecture  selection. 

•  Pre-existing  stand-alone,  stable  legacy  systems  generate  connection 
architecture  requirements. 

•  Workload  and  attention  management  models  spawn  task  and  task 
management  requirements  [3],  impacting  business  logic  design. 

•  Decision  support  and  supervisory  control  tools  generate  HCI 
requirements. 


HCI  LOGIC 

MCD  represents  a  radical  departure  from  the  famil¬ 
iar  real  estate  and  insurance  office  GUIs.  Such  com¬ 
mercial  systems  are  application-centric,  data-driven, 
and  present  the  document  metaphor  in  virtual  win¬ 
dows.  A  decision  tool-centric,  task-driven  HCI  has 
an  inherently  different  look  and  feel.  For  example,  in 
air  defense,  an  autolinking  GUI  function  displays 
amplifying  information  in  a  decision  tool  whenever  a 
map  track  is  selected  (Figure  1).  Autolinking  primi¬ 
tives  are  part  of  the  MCD  HCI  core  logic. 

MODELING 

A  user-centric  task  analysis  is  performed  on  an  oper¬ 
ational  domain  to  establish  its  essential  use  cases.  Use 
cases  capture  dynamic  aspects  of  a  system  and 
expose  critical  requirements.  The  identified  cases  are 
next  catalogued  into  tasking  families.  A  properly 
prepared  use  case  can  be  mapped  to  an  operational 
task  model.  Table  1  gives  sample  essential  cases  for 
three  task  families  in  the  air  defense  domain. 

Once  the  core  task  set  has  been 
identified,  a  task  model  is  devel¬ 
oped.  A  generic  task  is  a  goal- 
oriented  work  activity  that  results 
in  a  product  [1].  As  such,  it  con¬ 
tains  both  concrete  and  abstract 
elements.  Concrete  software 
activities  such  as  read-ahead  data 
caching  are  modeled  in  the  task. 

Abstract  elements  such  as  the 
cognitive  processes  involved  in 
making  decisions  cannot  be  mod¬ 
eled  in  software  and  are  not 
included.  They  remain  part  of  the 
overall  task,  however,  and  are 
explicitly  supported  in  the  HCI 
by  the  decision  support  tools. 


RESPONSE  ALTITUDE/ID 
HISTORY  HISTORY 

SQUARES  TRAIL 


DROP  RANGE  FROM  OWNSHIP  SPEED 
LINES  HISTORY 


TRAIL 


FIGURE  1 .  Track  Profile  at-a-glance  decision  support  tool.  Displays 
the  currently  hooked  vehicle’s  track,  including  its  altitude,  speed, 
and  response  history.  Implemented  in  Java™.  Integrated  through 
autolinking  with  other  HCI  components  using  Java  technologies. 


TABLE  1 .  Essential  air  defense  warfare  use  cases  factored  into  three  task  families. 


Monitor  Air  Situation 

Conduct  Intercept  and  Escort 

Respond  to  Air  Threat 

Issue  New  Track 

Verbal  Report 

Manage  DCA  Cover 

Issue  Level  1  Query 

Issue  Update  Track 
Verbal  Report 

Manage  DCA  Intercept 

Issue  Level  2  Warning 

Order  DCA  Action 

Illuminate  Track 

1 

Manage  DCA  Engage 

Order  Cover 

1 
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A  task  model  is  composed  of  a  set  of  task  super  classes  that  declare  com¬ 
mon  task  attributes  and  behaviors.  Domain-specific  task  functionality  is 
added  in  derived  concrete  classes.  This  approach  helps  to  decouple  task 
logic  from  HCI  logic,  making  the  task  classes  reusable.  A  task  model  can 
write  to  multiple  HCI  components,  providing  display  flexibility.  A  task 
model  can  also  have  its  own  GUI  set,  which  means  that  tasks  can  be 
stored  on  servers,  and  can  take  their  GUIs  along  when  they  move  to 
clients.  Figure  2  shows  a  task  class  component. 

Tasks  are  triggered  by  environmental  events  and  are  assigned  to  work¬ 
group  members.  The  assignment  logic  is  based  on  team  workload  and 
individual  capability  models  [4].  These  triggering  and  management  func¬ 
tions  are  deployed  in  the  architecture’s  application  tier  as  task  manage¬ 
ment  business  logic  components. 

Numerical  computations,  data  management,  and  hardware  control  func¬ 
tions  are  performed  by  the  legacy  system.  The  task  model  gets  structured 
information  from  the  system  "just-in-time"  and  presents  it  in  the  HCI  in 
the  form  of  results,  recommendations,  and  draft  products.  The  model 
directs  the  legacy  system,  through  a  connection  mechanism,  to  execute 
functions  and  provide  status,  drill-down,  and  environmental  information. 


TaskBnse 


+addTaskListener() 

+removeTaskListener() 

+getException() 

+getFullName() 

+getPriority() 

+getState() 

+isAlive 

+start() 

+suspend( ) 

+switchln() 

+switchOul() 

+terminate() 


FIGURE  2.  A  task  class  component. 


Figure  3  shows  a  task  manage¬ 
ment,  decision  support  conceptual 
model. 

APPROACH 

Web  and  emerging  enterprise 
technologies  map  well  to  MCD 
features  and  functions.  While 
many  new  technologies  are  still 
evolving,  others  are  mature 
enough  to  exploit  now.  New 
technology  use  is  often  asso¬ 
ciated  with  the  following  "best" 
practices: 


FIGURE  3.  MCD  task  management  conceptual  model. 


Design  for  the  enterprise.  The  def¬ 
inition  of  "enterprise"  scales  from 

combat  information  center  teams,  to  the  entire  ship,  to  the  battlegroup, 
and  beyond.  Large,  stable  legacy  systems,  which  should  be  left  undis¬ 
turbed,  form  the  enterprise’s  information  management  backbone. 
Enterprise  components  such  as  the  MCD  Task  Manager  tap  into  the 
ship’s  information  management  backbone  through  standard  connection 
mechanisms. 


Adopt  or  adapt  a  standard  software  architecture.  Industry  groups  are 
developing  architecture  standards.  Multi-tiered  architectures  enable 
developers  to  separate  data  display  from  data  processing  and  data  storage. 
Benefits  include  code  reuse,  scalability,  and  ease  of  integration. 

Focus  on  writing  business  logic  while  leveraging  commercial  and  open 
source  "enabling"  technologies.  Application  servers  provide  multi-user 
access  to  shipboard  planning  and  tactical  components.  The  MCD  man¬ 
agement  components  are  served  across  the  enterprise.  This  means  that 
personnel  not  stationed  at  consoles  have  access  to  the  same  information 
as  the  watchstanders. 
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Maximize  the  use  of  standard  communication  protocols.  MCD’s  adapter 
tier  accesses  legacy  information  management  components  through  native 
protocols,  and  publishes  to  clients  over  high-level,  standard  protocols 
such  as  Simple  Object  Access  Protocol  (SOAP).  This  strategy  helps  to 
reduce  code  life-cycle  costs. 


MCD  ARCHITECTURE 


A  composite  system’s  architectural  form  can  be  allowed  to  accrete  over 
time  as  disparate  legacy  systems  are  interconnected.  The  final  architecture 
is  defined  after  all  the  connections  are  realized.  This  process  produces  the 
familiar,  and  yet  always  puzzling,  lines  and  boxes  architecture.  An  alter¬ 
native  is  to  specify  the  architecture  as  soon  as  the  essential  use  cases  have 
been  defined,  their  requirements  identified,  and  the  conceptual  models 
blocked  out.  Example  candidate  architectures  include  the  blackboard 
model,  hub  and  spoke,  client/server,  n-tiered,  International  Organization 
for  Standardization  (ISO)  network  layer  model,  and  neural  net. 

MCD  has  workgroup  requirements  and  legacy  system  connectivity 
requirements.  Current  best  practices  suggest  adopting  an  n-tiered  archi¬ 
tectural  approach  [4].  The  MCD  task  and  workload  management  compo¬ 
nents  represent  "business  logic"  and  belong  in  the  application  tier.  HCI 
display  components  properly  reside  in  the  presentation  tier.  Legacy  sys¬ 
tem  adapters  and  Web  services  make  up  the  connection  sub-architecture. 
The  corporate  system  is  its  own  architecture,  and  makes  up  the 
legacy  tier. 


PRESENTATION  TIER 


WEB-ENABLED 
HCI  COMPONENT 


THICK 

HCI  COMPONENT 


CLIENT  TASK 
MANAGER 


CLIENT  WORKLOAD 
MONITOR 


TASKS 


Figure  4  depicts  MCD  deployed  as  a  distributed,  task  management  enter¬ 
prise  application.  Presentation  components  are  both  thin  and  thick,  as 
dictated  by  HCI  autolinking 
requirements.  Thin  components 
run  on  watchstations,  on  laptops, 
and  may  eventually  run  on  per¬ 
sonal  digital  assistants  (PDAs). 

The  Task  and  Workload  Managers 
are  transaction-capable,  and  are 
implemented  as  Enterprise  Java 
Beans  ™  [5].  Task  templates  are 
stored  in  local  databases  and  are 
instantiated  on  receipt  of  external 
events  such  as  a  call-for-fire. 

Instantiated  tasks  execute  on 
clients,  mediating  the  presentation 
of  data,  status,  and  draft  products 
in  the  HCI.  The  connection  tier 
"talks"  native  protocol  on  the 
legacy  side  and  standard  Web 
services  on  the  task  management 
side.  The  legacy  system’s  public 
functions  have  been  exposed 
through  a  variety  of  means, 
including  native  application  pro¬ 
gram  interfaces  and  Common 
Object  Request  Broker 
Architecture  (CORBA®). 
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FIGURE  4.  Conceptual  model  of  an  n-tiered  task  management  system.  The  user-centered 
HCI  contains  decision  support  and  supervisory  control  GUI  elements.  The  Presentation 
and  Application  tiers  connect  to  corporate  information  management  systems  via  Web 
services  and  custom  adapters. 
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CONCLUSIONS 

Building  a  user-centered  task  visualization  HCI  is  a  novel  undertaking.  It 
is  also  a  challenging  proposition  because  traditional  GUI  development 
approaches  and  tools  are  not  flexible  and  powerful  enough.  The  answer  is 
to  exploit  new  enterprise  technologies,  combine  them  with  object  model¬ 
ing  methodologies,  and  drive  the  development  effort  with  requirements 
derived  from  user  task  analyses.  The  resulting  product  will  be  a  user- 
centered  task  management  system  that  can  be  connected  to  a  variety  of 
legacy  applications. 
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INTRODUCTION 

The  Real-time  Execution  Decision  Support  (REDS)  program  is  develop¬ 
ing  automated  force-level  decision  aids,  or  software  tools,  that  facilitate 
situational  awareness,  risk  assessment,  and  responsive  retargeting  for 
naval  air  strike  assets  such  as  those  shown  in  Figure  1.  These  decision 
tools,  being  developed  as  an  integrated  Decision  Support  Suite  (DSS) 
referred  to  as  the  REDS-DSS  effort,  include  three  major  tools.  The 
Sensors,  Intelligence,  ROEs  (rules  of  engagement),  and  Environment 
Network  (SIREN)  module  supports  situation  awareness.  The  Risk 
Assessment  and  Validation  Engine  (RAVE)  provides  risk  assessment 
capability.  The  Rapid  Asset  Pairing  Tool  (RAPT)  provides  responsive 
targeting  options  generation  to  the  kill  authority. 

Under  development  as  an  Office  of  Naval  Research  (ONR)  Knowledge 
Superiority  and  Assurance  (KSA)  Future  Naval  Capability  (FNC),  these 
tools  are  slated  for  transition  to  the  Joint  Mission  Planning  System 
(JMPS)  force-level  release. 

The  dynamic  arena  in  which  tactical  air  assets  operate  dictates  the  need 
for  flexible  targeting  options  generation  in  the  event  of  significant 
changes  in  the  environment,  introduction  of  high-priority  targets,  or  tar¬ 
gets  of  opportunity.  The  ability  to  generate  weapon-target  pairings  is 
complicated  by  the  incorporation  of  advanced  munitions. 

An  automated  force-level  planning  tool  must  provide  automated  flight¬ 
planning  and  mission-planning  capabilities  to  the  tasking  authority. 
Requirements  for  an  automated  flight-planning  tool  include  automated 
routing,  platform  capabilities  and  performance  data  pulls,  and  weather 
forecasts.  Aspects  of  an  automated  mission-planning  tool  include  the 
ability  to  coordinate  disparate  aircraft,  sensor,  and  weapon  systems, 
deconflict  routes,  and  evaluate  mission  effectiveness  and  platform  risk.  In 
concert  with  the  commander’s  intent,  this  tool  should  also  support  auto¬ 
mated  retasking  and  asset  allocation  [1], 

The  REDS-DSS  tools  allow  a  tasking  authority  to  respond  to  dynamical¬ 
ly  changing  targeting  situations  using  in-theater  assets.  The  near-term 
decision  support  objective  is  to  move  an  air-strike  package  en  masse  to 
nearby  high-priority  targets  as  well  as  react  to  changes  in  the  environ¬ 
ment.  The  long-term  objective  of  this  work  is  to  provide  a  capability  to 
retarget  strike  assets  beyond  the  original  strike  area  with  the  potential 
inclusion  of  joint  and/or  allied  assets. 


ABSTRACT 

The  Real-time  Execution 
Decision  Support  (REDS )  pro¬ 
gram  is  developing  software 
tools  that  support  air  strike 
mission  planning  and  dynamic 
replanning  for  time-critical 
operations.  These  tools  are 
hosted  on  an  enterprise  archi¬ 
tecture  that  integrates  distrib¬ 
uted  computing  technologies  to 
yield  seamless  transitions  from 
planning  to  real-time  mission 
execution  and  monitoring 
while  supporting  tailored 
information  transfer  among 
planning  and  operational  ele¬ 
ments.  Thus,  the  warfighter 
realizes  an  increased  ability  to 
respond  to  changing  battlefield 
and  operational  conditions.  In 
this  state-of-the-art  concept, 
real-time  mission  repair, 
replanning,  and  retargeting  of 
in-theater  assets  is  achieved. 
The  REDS  Dynamic  Decision 
Support  Suite  serves  as  a  force- 
level  support  tool  for  real-time 
retargeting  by  providing  infra¬ 
structure  and  algorithms  for 
situation  awareness,  risk  assess¬ 
ment,  and  near-optimal  strike 
asset  allocation. 


FIGURE  1 .  The  REDS-DSS  provides 
decision  support  for  naval  air  strike. 
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SITUATION  AWARENESS 

The  cornerstone  of  time-critical  strike  resides  in  rapidly  assessing  the 
common  operational  picture  (COP)  in  light  of  mission  objectives.  The 
REDS-DSS  accomplishes  this  by  pulling  forward  track  information  and 
associated  system  capabilities,  performance  data,  and  mission  plans.  The 
SIREN  module  has  been  developed  in  an  effort  to  meet  this  requirement. 
SIREN’s  objectives  include: 

•  Monitoring  the  environment 

•  Assessing  changes  to  the  entities  in  the  environment 

•  Identifying  entities  in  the  environment 

•  Retrieving  entity  capabilities  and  performance  information 

•  Managing  the  prioritized  target  list 

•  Retrieving  mission  data 

Interfaces  to  a  multitude  of  systems  enable  the  SIREN  module  to  pull 
forward  the  capabilities  and  performance  of  entities  in  the  COP.  A 
Global  Command  and  Control  System-Maritime  (GCCS-M)  feed  pro¬ 
vides  track  information  complete  with  Red  Force  electronic  intelligence 
notation  (ELNOT).  The  ELNOT  number  is  passed  to  the  EA-6B 
Tactical  Information  and  Report  Management  System  (ETIRMS)  to 
obtain  a  system  designator.  The  system  designator  is  then  passed  to 
Quiver,  a  Red  Force  Threat  and  Target  database,  which  returns  the  Basic 
Encyclopedia  (BE)  number,  location,  time  stamp,  and  corresponding 
capabilities  and  performance  parameters  of  the  threat  system.  In  the  event 
of  unknowns,  a  composite  track  correlator  capability  estimates  the  entity 
type  based  on  the  track  dynamics  and  any  associated  intelligence  infor¬ 
mation. 
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In  addition  to  supporting  Blue  Force  capability  and  performance  data 
from  the  Joint  Mission  Planning  System,  SIREN  provides  an  interface  to 
the  Element  Level  Planning  (ELP)  Tool/Strike  Planning  Folder  to  com¬ 
bine  mission-planning  informa¬ 
tion  with  corresponding  Blue 
Force  track  data  to  form  a  com¬ 
prehensive  data  object  for  risk 
assessment  and  dynamic  asset 
allocation. 

As  new  entities  (or  tracks)  enter 
the  specified  region  of  interest, 
they  appear  on  the  Gantt  timeline 
based  on  their  time  stamp  (Time- 
in-COP,  or  TIC)  and  a  textual 
window  alerts  the  user  that  a  new 
entity  has  entered  the  COP. 

When  the  entity  is  no  longer  rep¬ 
resented  by  a  track,  it  leaves  the 
COP  (Time-out-COP,  or  TOC), 
terminating  the  Gantt  timeline. 

Trends  can  be  readily  visualized 
via  the  Gantt  bars  as  entities 
become  active  and  inactive  within 
the  COP  based  on  their  TIC- 
TOC  indicators.  Figure  2  shows  a 
prototype  of  the  SIREN  Client 
Viewer. 
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FIGURE  2.  The  REDS-DSS  Client  Viewer  supports  a  Gantt  presentation  of  when  entities 
enter  and  leave  the  COP  as  well  as  a  map  interface  and  alert  window.  This  is  a  view  of  the 
SIREN  module  interface.  Tabs  are  provided  to  access  other  modules. 
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A  map  viewer  is  provided  as  part  of  the  SIREN  user  interface.  This  view¬ 
er  can  accommodate  user  preferences.  For  example,  the  user  can  specify 
the  entities  to  monitor  by  selecting  them  from  a  list  of  entities  in  the 
COR  An  intelligence  officer  can  display  the  enemy  order  of  battle  and 
keep  the  watchstander  abreast  of  any  changes  that  may  affect  the  mission. 
The  airboss  can  watch  the  execution  of  the  airplan  when  entities  under 
his  purview  execute  their  mission. 

Additional  viewer  functionality  includes  the  ability  to  toggle  entity  fades 
in  the  map  as  a  function  of  the  latency  of  the  data  and  system  type.  Thus, 
the  user  can  readily  assess  the  latency  of  the  track  information.  The  user 
can  also  specify  regions  of  interest  such  as  no-fly  zones  or  air  corridors 
that  can  be  used  to  generate  alerts  should  an  entity  enter,  or  leave,  a  speci¬ 
fied  volume. 

Finally,  a  prioritized  target  list  management  function  is  provided  to  sup¬ 
port  the  insertion  of  high-priority  and/or  time-critical  targets.  Existing 
targets  are  taken  from  the  Joint  Integrated  Prioritized  Target  Fist 
(JIPTF);  the  list  is  augmented  as  necessary  with  the  insertion  of  targets  of 
opportunity.  This  insertion  of  high-priority  targets  or  targets  that  have 
short  windows  of  opportunity  has  the  potential  to  initiate  the  RAPT  to 
generate  new  weapon-target  pairing  options. 

RISK  ASSESSMENT 

The  evolutionary  threat  arena  requires  the  constant  monitoring  of  threats 
to  all  Blue  Force  entities  in  the  battlespace.  This  capability  is  paramount 
in  triggering  a  dynamic  retargeting  event.  RAVE  is  being  developed  to 
support  the  following  functionality: 

•  Determining  risk  to  entities  in  the  COP 

•  Providing  a  risk-based  trigger  function  to  the  dynamic  retasking  tool 

•  Validating  threat  capability  based  on  situational  information 

•  Providing  a  mechanism  for  displaying  risk  evaluations  in  real  time 

•  Quantifying  deconfliction  as  part  of  risk  assessment 

The  methods  implemented  for  ascertaining  risk  to  a  Blue  Force  entity 
depend  on  the  entity  state.  A  static  entity  can  be  evaluated  based  on  its 
current  position  and  proximity  to  red  forces  and  Red  Force  capabilities. 

A  mobile  entity  must  be  proactively  evaluated  based  on  its  planned  route 
or  perceived  path. 

For  Strike- Air  Assets,  the  RAVE  tool  uses  kill-chain  analysis  to  quantify 
and  assess  platform  risk  based  upon  the  threat  laydown.  Templates  of  the 
radar  signature  for  tactical  aircraft  of  interest  have  been  generated  against 
various  threats.  These  templates  are  used  for  determining  the  level  of 
exposure  that  an  aircraft  has  against  a  particular  threat  system  based  on 
the  aircraft’s  orientation,  altitude,  and  slant  range  from  the  threat.  The 
technique  for  estimating  the  threat-based  risk  component  to  an  airborne 
asset  is  a  function  of  exposure  time  and  threat  capabilities.  The  route  is 
decomposed  into  1-second  intervals  based  on  flight  performance  parame¬ 
ters  and  then  evaluated  to  determine  if  the  route  points  are  within  threat 
range.  The  continuous  time  that  a  platform  is  within  threat  range  is  then 
integrated  and  compared  with  a  risk-level  threshold.  When  the  platform 
is  no  longer  within  tracking  or  acquisition  range,  the  risk  level  can  be 
zeroed  out  based  on  the  required  time  of  continuous  non-exposure  to  the 
threat  to  break  the  radar  lock.  Risk  from  multiple  threats  on  a  single  plat¬ 
form  is  assessed  using  the  Flamacher  sum  as  discussed  by  Sugianto  [2]. 
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For  the  current  implementation,  the  /°°  norm  can  be 
taken  as  the  overall  risk  of  the  route. 

An  example  illustrating  this  concept  is  shown  in 
Figures  3  and  4.  Figure  3  shows  a  hypothetical  route 
and  crossing  through  a  stationary  (threat-centric) 
template.  For  this  example,  the  template  does  not 
change  with  respect  to  aircraft  orientation,  altitude, 
or  range.  Figure  4  shows  the  results  of  evaluating 
the  route  through  the  threat  region  based  on  kill- 
chain  analysis.  The  top  portion  of  Figure  4  indicates 
when  the  aircraft  is  within  range  of  the  threat’s 
radar.  The  middle  portion  of  Figure  4  shows  the 
evaluation  of  the  continuous  time  of  exposure  rela¬ 
tive  to  the  risk  threshold  (shown  by  the  dashed 
line).  The  bottom  part  of  Figure  4  shows  the  nor¬ 
malized  risk  along  the  route. 

RESPONSIVE  TARGETING 

RAPT  supports  the  dynamic  reallocation  of  strike 
force  assets.  While  the  optimization  algorithms  are 
generic,  the  focus  is  on  naval  air  strike  missions  [3]. 

The  objectives  for  the  RAPT  module  include: 

•  Rapid  weapon-target  pairing  options  generation 
to  tasking  authority 

•  Providing  an  automatic  routing  capability 

•  Dynamically  reallocating  assets  based  on  high- 
priority  target  trigger 

•  Dynamically  reallocating  assets  based  on  chang¬ 
ing  environment 

•  Minimizing  risk  and  collateral  damage  while 
maximizing  effectiveness 

The  overarching  goal  of  the  RAPT  is  to  provide 
weapon-target  pairing  recommendations  to  a  com¬ 
mander  with  tasking  authority.  The  RAPT  can  be 
triggered  based  on  the  insertion  of  a  high-priority 
target  into  the  mission  objectives  as  well  as  excessive 
risk  levels  of  threats  impinging  on  the  Blue  Force 
assets.  An  example  of  the  latter  is  shown  in  Figure  5. 

As  shown,  a  platform  launches  with  an  inherent  risk 
level.  If  environment  changes  adversely,  affecting 
the  risk  level  of  a  platform,  the  RAPT  is  triggered  to 
generate  weapon-target  pairing  recommendations  in 
a  continuous  manner  to  facilitate  options  generation  at  some  time  T. 
However,  weapon-target  pairings  do  not  constitute  a  complete  solution. 
Route  information,  and  the  risk  assessment  and  deconfliction  of  that 
route,  must  also  be  provided  with  any  recommended  weapon-target  pair¬ 
ings  to  form  a  complete  solution.  These  options  are  presented  to  the  task¬ 
ing  authority  for  their  modification  and/or  approval. 

The  formulation  of  the  objective  function  for  the  strike  asset  optimiza¬ 
tion  problem  is  given  by  McDonnell  et  al  [3].  The  optimization  engine 
has  been  chosen  based  on  the  need  to  support  continuous  options  genera¬ 
tion,  a  readily  modifiable  objective  function,  and  the  potential  to  generate 
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FIGURE  3.  A  hypothetical  example  of  the  exposure  template  used 
to  assess  the  risk  to  positions  and  routes  through  enemy  threat 
laydown. 
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FIGURE  4.  Quantification  of  the  RAVE  kill-chain  analysis  for 
assessing  risk. 
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global  solutions  on  a  multimodal  surface  within  a 
myriad  of  constraints.  The  optimizer  chosen  is  based 
on  evolutionary  computation  techniques  augmented 
with  case-based  reasoning  methods.  Preliminary 
studies  [4]  have  shown  that  the  optimizer  works 
rapidly  for  relatively  small  problem  sets  such  as  that 
shown  in  Figure  6.  Allocation  of  20  platforms  with 
80  assets  attacking  20  targets  can  be  performed  in 
seconds  with  a  relatively  short  number  of  iterations 
as  shown  in  Figure  7. 

While  the  convergence  speed  of  these  results  is 
encouraging,  additional  work  needs  to  be  done  to 
address  allocating  assets  under  scheduling  con¬ 
straints.  Namely,  the  choreography  of  a  naval  strike 
mission  revolves  around  time-on-target,  Suppression 
of  Enemy  Air  Defenses  (SEAD)  timings,  and  tank¬ 
ing  constraints.  To  help  decouple  the  problem,  the 
search  is  partitioned  such  that  attack  assets  are  only 
allocated  to  attack  targets,  while  SEAD  assets  are 
only  allocated  to  SEAD  targets. 

A  tradeoff  exists  between  maximizing  mission  effec¬ 
tiveness  and  minimizing  overall  risk  to  the  mission. 

A  slider  that  emphasizes  the  effectiveness /risk  tradeoff 
is  provided  to  the  tasking  authority  to  incorporate 
the  commander’s  intent  into  the  optimization  engine. 
Another  objective  that  is  being  incorporated  is  "per¬ 
sisting"  the  airplan  [5].  That  is,  changes  to  the  existing 
missions  should  be  minimized  to  prevent  wholesale 
modifications  of  the  planned  missions. 


CHANGES  STRIKE 

ASSETS 


FIGURE  5.  A  notional  depiction  of  increased  risk  providing  a  trigger 
to  the  RAPT  engine  based  on  environmental  changes  and  subsequent 
optimization  to  reduce  the  exposure. 


FIGURE  6.  RAPT  provides  dynamic  weapon-target  pairing  for 
many-on-many  assets  on  objectives. 


FIGURE  7.  An  evolutionary  algorithm  can  generate  near-optimal 
weapon-target  pairings  in  a  matter  of  seconds. 
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CONCLUSION 

The  REDS-DSS  tools  will  provide  the  warfighter  with  an  increased  ability 
to  respond  to  changing  battlefield  and  operational  conditions  while  main¬ 
taining  operational  tempo.  With  this  suite  of  tools,  mission  repair,  replan¬ 
ning,  and  retargeting  of  in-theater  assets  may  be  achieved  in  response  to 
high-priority  targets  and  intelligence  updates. 

The  REDS-DSS  tools  are  hosted  on  a  J2EE  enterprise  architecture.  This 
architecture  is  open  and  extensible  and  has  interfaces  to  a  multitude  of 
legacy  systems  such  as  the  Joint  Munitions  Effectiveness  Manual  (JMEM) 
and  data  sources  such  as  GCCS-M.  In  addition,  Enterprise  Java  Beans  are 
being  employed  so  that  interfaces  with  other  legacy  applications  such  as 
the  Portable  Flight  Planning  System  (PFPS)  and  new  services  can  be 
readily  provided  as  they  become  available  [6]. 
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Rapidly  Expanding  World  of  C4I 

Penney  Myer  and  Sue  Patterson 

SSC  San  Diego 


INTRODUCTION 

The  U.S.  military  today  and  in  the  future  will  conduct  operations  within 
the  joint,  allied  and  coalition  environments  simultaneously.  Within  this 
context,  the  ability  to  exchange  information  across  multiple  security  lev¬ 
els  has  become  an  even  greater  necessity.  To  effectively  use  today’s  preci¬ 
sion  weapons  and  ensure  a  common  battlefield  picture,  our  allied  and 
coalition  partners  must  be  fully  integrated  in  and  interoperable  with  our 
command,  control,  communications,  computers,  and  intelligence  (C4I) 
environment.  The  U.S.  must  be  able  to  operate  in  an  environment  where 
the  data  flow  rapidly  and  seamlessly  between  the  Top  Secret  sensitive 
compartmented  information  (SCI),  general  services  (GENSER),  and 
coalition  participants.  This  architecture  must  provide  the  ability  to 
acquire,  store,  and  disseminate  content  to  and  from  these  diverse  security 
domains.  In  addition,  it  must  allow  users  with  varying  clearances  to 
securely  collaborate  with  one  another  using  both  structured  and  unstruc¬ 
tured  methods.  The  architecture  must  demonstrate  that  it  can  accommo¬ 
date  these  features  in  an  approach  that  is  scalable  (from  a  hardware  per¬ 
spective),  supportable  (from  a  manpower  perspective),  and  able  to  be 
implemented  (from  an  engineering,  security,  and  acquisition  perspective). 

The  Navy’s  Ocean  Surveillance  Information  System  (OSIS)  Evolutionary 
Development  (OED)  is  the  only  system  that  is  fully  accredited  for  multi¬ 
level  secure  (MLS)  processing  by  the  Department  of  Defense  (DoD).  It 
was  first  certified  and  accredited  for  operational  use  in  September  1998 
and  has  been  operating  at  U.S.  joint  intelligence  centers  and  foreign 
national  intelligence  centers  since  then. 

OED  evolved  out  of  the  OSIS  program  of  the  1970s.  Initially,  OSIS  was 
established  to  support  decision-makers  at  all  levels  of  command.  OSIS 
emerged  as  a  combination  of  personnel,  facilities,  computers,  communi¬ 
cations,  and  procedures  designed  to  receive,  process,  correlate,  and  dis¬ 
seminate  evaluated  land,  air,  and  ocean  surveillance  information.  In  1997, 
the  OSIS  Baseline  Upgrade  system  became  known  as  OED,  which  now 
serves  as  the  backbone  automated  information  fusion  system  currently 
supporting  a  multilevel  common  operational  picture  at  U.S.  and  foreign 
intelligence  centers.  There  has  also  been  significant  interest  in  leveraging 
OED’s  MLS  and  intelligence  capabilities  by  the  operational  fleet. 
Consequently,  OED  is  currently  undergoing  test  and  evaluation  for  the 
afloat  community. 


ABSTRACT 

This  paper  describes  the  MLS 
solutions  developed  by  the 
Ocean  Surveillance 
Information  System  (OSIS) 
Evolutionary  Development 
(OED)  Project.  OED’s  design 
is  based  on  the  tight  coupling 
of  a  commercial  trusted  oper¬ 
ating  system  as  its  foundation 
and  a  large  amount  of  govern¬ 
ment  and  commercial  single- 
level  applications  software, 
with  a  small  amount  of  multi¬ 
level  trusted  application  soft¬ 
ware.  OED ’s  multilevel 
approach  implements  a  process 
where  data’s  original  security 
domains  are  preserved 
throughout  the  analysis  and 
fusion  process.  Information  is 
automatically  released  accord¬ 
ing  to  its  original  security 
domain,  and  fused  intelligence 
is  released  according  to  the 
sum  of  its  security  domain. 
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OED  operates  at  the  Top  Secret/SCI  level;  its  design  is  based  on  a  com¬ 
mercial  trusted  operating  system  (Hewlett-Packard  Unix/Trusted 
Operating  System  [HP-UX/TOS]  10.26)  and  a  combination  of  govern¬ 
ment  and  commercial  single-level  applications  software. 

This  paper  describes  the  high-level  design,  implementation,  and  deploy¬ 
ment  of  OED. 

MLS  DESIGN 

MLS  is  a  complex  concept  and  a  highly  specialized  area  of  computer 
security.  MLS  means  different  things  to  different  users: 

•  External  MLS  -  the  ability  to  take  information/intelligence  from  mul¬ 
tiple,  trusted  security  levels  and  disseminate  derived  products  out  to 
multiple,  trusted  security  levels. 

•  Internal  MLS  -  the  ability  to  restrict  access  to  data  on  a  network 
depending  on  the  security  level  of  the  user. 

•  MLS  communications  -  systems  that  can  take  message  traffic  from 
multiple  communications  paths  and  transmit  messages  out  via  multiple 
communications  paths. 

•  MLS  C4I  -  a  system  that  can  take  data  from  multiple  security  levels 
and  preserve  the  data’s  security  label  as  it  moves  through  the  analysis, 
fusion,  and  dissemination  process  with  a  high  level  of  assurance.  This 
assurance  means  that  the  data  processing  is  in  accordance  with  an 
approved  security  policy. 

MLS  is  often  confused  with  the  concept  of  multiple  security  levels 
(MSL).  MSL  is  a  class  of  system  composed  of  relatively  untrustworthy 
single-level  systems.  Separation  of  data  and  trust  is  placed  in  controlled 
interfaces  between  the  less  trustworthy  components.  These  controlled 
interfaces  (e.g.,  guards  and  sanitizers)  are  typically  smaller  automated 
information  systems  running  a  dedicated  program,  providing  a  dedicated 
function. 

MSL  implementations  maintain  multiple  instantiations  of  servers,  clients, 
applications,  and  databases  to  serve  each  security  enclave  and  pass  the 
data  up  or  down  using  guard  and  sanitization  tools.  Because  of  this,  it  is 
not  uncommon  to  find  three  or  more  workstations  in  a  single  workspace 
aboard  our  ships. 

OED  is  DoD’s  only  accredited  multilevel  Protection  Level  4  (PL4)  secure 
intelligence  processing  and  data  dissemination  system.  It  is  certified  and 
accredited  by  the  Special  Security  Office  Navy  and  the  Defense 
Intelligence  Agency.  It  serves  as  the  backbone  automated 
information/fusion  system  supporting  a  multilevel  common  operational 
picture  at  U.S.  and  allied  joint  intelligence  centers.  OED  typically  oper¬ 
ates  at  the  Top  Secret/SCI  level  and  automatically  and  manually  dissemi¬ 
nates  formatted  and  narrative  messages  at  multiple  security  levels.  OED’s 
most  current  Release  4. 0.4. 3  software  and  hardware  has  been  certified  for 
PL2  (formerly  known  as  compartmented  mode)  for  internal  communica¬ 
tions  and  PL4  (formerly  known  as  multilevel  mode  or  MLS)  for  external 
operations. 

The  system  can  automatically  receive  and  generate  tens  of  thousands  of 
narrative  and  formatted  messages  per  day,  from  a  variety  of  different 
security  domains  and  communications  sources  (these  include  Officer  in 
Tactical  Command  Information  Exchange  Subsystem,  Tactical  Data 
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Information  Exchange  Subsystem,  and  Defense  Message  System).  The 
system  is  also  directly  connected  to  the  Automatic  Digital  Network 
(AUTODIN)  Bypass  systems  (i.e.,  Communication  System  Processor 
and  Newsdealer)  for  connectivity  with  allies,  coalition  partners,  and 
worldwide  SCI  sensors  and  command  centers.  OED’s  MLS  communica¬ 
tions  subsystems  ensure  rapid 
delivery  of  both  record  message 
traffic  and  intelligence  broadcasts 
in  support  of  the  Unified 
Commanders-in-Chief,  Joint  Task 
Force  commanders,  and  coalition 
warfare  partners.  OED’s  accredit¬ 
ed  MLS  design  allows  simultane¬ 
ous  data  outputs  to  a  number  of 
security  levels.  The  typical  "low¬ 
est  common  level"  problem  in 
coalition  warfare  is  completely 
avoided.  See  Figure  1. 

The  information  flow  of  the 
OED  system  involves  input, 
internal  processing,  and  output 
capabilities.  Inputs  into  the  sys¬ 
tem  arrive  from  external  sources 
in  the  form  of  messages  via  com¬ 
munications  circuits  and  data 
directly  entered  by  the  user. 

These  messages  are  received  as 
either  formatted  messages,  which 
can  be  automatically  processed  by 
the  system  producing  formatted 
output  messages,  or  as  narrative 
messages  (i.e.,  unformatted  mes¬ 
sages).  Unformatted  or  narrative 
messages  cannot  be  automatically 
processed  for  output;  however, 
every  incoming  message  is  auto¬ 
matically  and  rigorously  validated 
for  overall  integrity,  and  a  securi¬ 
ty  label  is  automatically  attached 
to  the  data.  Only  formatted  mes¬ 
sages  with  valid  security  labels  are 
allowed  to  contribute  to  the 
labeled  track  data  picture.  See 
Figure  2. 

OED  software  does  more  than 
support  formatted  and  narrative 
message  processing.  OED  appli¬ 
cations  include: 

•  Robust  message  profiling/ 
retrospective  searches. 

•  Unique  data-mining  tools  (with 
the  Contiguous  Connection 
Module). 


PHASE  III  CONNECTIVITY  CAPABILITY  (IOC  Q4  '02) 
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FIGURE  1 .  Phase  ITT  connectivity. 


RIGID  DATA  LABELING  POLICY  PERMITS  MULTILEVEL  ACCREDITATION/USES 


EXTERNAL  DATA  INPUTS 


MANUAL  INPUT/EDITS 


ALL  INCOMING  DATA  MUST  BE  LABELED  BEFORE  ENTERING  ANY  OED  DATABASE 

LABELS  ARE  RETAINED  BY  OBJECTS  THROUGHOUT  THEIR  LIFE  IN  THE  SYSTEM 

SECURE  O.S  RENFORCES  ALL  DATA  AND  APPLICATION  ACCESS  (BOTH  INTERNAL  AND  EXTERNAL) 


FIGURE  2.  Rigid  data  labeling  policy. 
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•  Automated  correlation  of  incoming  contacts  to  tracks  in  a  single 
multilevel  track  database  with  significantly  improved  track  history 
for  analytical  use. 

•  Multiple  databases  that  contain  information  to  assist  the  intelligence 
analyst  in  correlation  and  analysis. 

All  OED  equipment  resides  in  secure  spaces.  Due  to  physical  security 
issues,  all  personnel  granted  access  to  the  area  must  be  cleared  to  the 
highest  level  of  the  data  processed  by  the  OED  system.  All  work¬ 
stations/servers  are  interconnected  via  a  local-area  network  (LAN), 
which  uses  HP  10.26  Compartmented  Mode  Workstation  (CMW) 
operating  systems  and  multilevel,  labeled,  high-speed  network  com¬ 
munications  between  the  systems. 

All  OED  workstations  (HP  TACO,  TAC-4,  and  TAC-4  follow-on 
computer  systems)  and  servers  use  the  trusted  operating  system  with 
a  common  data  encoding  file  that  is  used  to  generate  and  validate  all 
security  labels  for  files,  messages,  and  processes  in  the  system.  Separate 
network  interface  cards  are  used  to  control  data  sent  to  and  received 
from  each  network  level.  The  OED  central  processing  unit  consists  of 
one  HP  CMW  server  (9000/J5000  or  J6000)  connected  via  Ethernet 
LAN  to  several  HP  Model  J5000  (or  B2600)  client  workstations  on 
the  SCI  High  LAN.  All  systems  support  one  or  two  19-inch  or 
20-inch  color  monitors.  All  systems  use  the  HP-UX/CMW  10.26 
secure  operating  system. 

MLS  IMPLEMENTATION/DEPLOYMENT 

OED  Release  4. 0.4. 3  software  has  been  installed  and  certified  at  a  num¬ 
ber  of  U.S.  and  allied  intelligence  sites.  OED  is  a  mission-critical  sys¬ 
tem  at  the  three  operational  U.S.  joint  intelligence  centers  (Joint  Forces 
Intelligence  Center,  Joint  Analysis  Center  Molesworth,  and  Joint 
Intelligence  Center  Pacific).  OED  is  also  the  central  intelligence  analysis 
system  at  four  foreign  national-level  intelligence  centers  (London, 
England;  Sydney,  Australia;  Funakoshi,  Japan;  and  multiple  sites  for  the 
Republic  of  Korea  Navy).  In  addition,  OED  is  sited  at  six  smaller  U.S. 
government  sites,  and  most  recently  aboard  the  Commander  Second 
Fleet  (C2F)  Flagship,  USS  Mount  Whitney  (LCC  20),  the  program’s 
first  afloat  installation.  See  Figure  3.  OED  was  used  operationally  dur¬ 
ing  NATO  Exercise  Strong  Resolve  2002,  Joint  Fleet  Exercise  02-02, 
as  well  as  during  daily  intelligence  watch  and  analytical  operations  in 
support  of  the  C2F  mission.  OED  responsibilities  on  the  ship  included 
support  for  U.S.  SCI  and  NATO  operators  within  the  dual  physical 
confines  of  the  commander’s  joint  intelligence  center.  OED  capabilities 
used  included  the  message-handling  system,  multilevel  common  opera¬ 
tional  picture,  and  data  mining.  In  addition,  OED  provided  a  consoli¬ 
dated  messaging  architecture  that  allowed  direct  connectivity  to  all 
security  domains  for  receipt,  archive,  and  generation  of  message  traffic. 
Connectivity  to  the  Allied  Information  Flow  System  (AIFS)  was 
achieved  during  initial  accreditation.  OED  was  assessed  by  C2F  as 
operationally  effective  with  significant  potential  to  be  the  Navy’s  intel¬ 
ligence  core  system  for  a  robust,  collaborative,  information  and  analy¬ 
sis  environment  afloat.  A  similar  system  is  scheduled  for  deployment 
aboard  USS  Blue  Ridge  (LCC  19)  in  support  of  Commander  Seventh 
Fleet. 
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OED  DATA  PATHS  ABOARD  USS  MOUNT  WHITNEY  (LCC  20) 


I _ 

FIGURE  3.  OED  data  paths. 

CONCLUSION 

The  War  on  Terrorism  has  highlighted  the  need  to  effectively  and  effi¬ 
ciently  move  data  across  a  wide  variety  of  security  domains.  This  require¬ 
ment  exists  not  only  in  the  area  of  allied  and  coalition  warfare  but  in  the 
area  of  homeland  security  and  defense  as  well.  The  events  of  September 
11th  showed  the  need  to  move  relevant  data  across  military  and  law 
enforcement  domains  as  well. 

To  meet  the  diverse  information  needs  of  joint,  allied,  and  coalition  part¬ 
ners,  the  DoD  has  implemented  MSL  solutions  both  ashore  and  afloat. 
These  implementations  have  resulted  in  multiple  instantiations  of  servers, 
clients,  applications,  and  databases  to  serve  each  of  the  diverse  security 
enclaves.  This  architecture  has  created  an  environment  that  hampers 
interoperability,  impedes  data  fusion,  creates  multiple  (and  divergent) 
common  operational  pictures,  results  in  data  loss,  and  creates  unnecessary 
duplication  of  hardware  (increased  space,  power  and  networks)  and  a 
concomitant  increase  in  operations  and  maintenance  costs. 

The  OED  system  has  untapped  potential  to  be  the  core  intelligence  sys¬ 
tem  both  ashore  and  afloat  for  the  U.S.  and  its  allied  partners.  OED’s 
MLS  features  can  serve  as  the  core  technology  on  which  future  C4I  sys¬ 
tems,  networks,  and  databases  operating  at  multiple  classification  levels 
can  be  effectively  accredited  and  combined.  As  the  system  evolves,  it  will 
allow  appropriately  cleared  analysts  and  watch  officer’s  high-speed  access 
to  information  across  multiple  security  domains. 
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INTRODUCTION 

Just  after  the  11  September  2001  attacks,  an  "unemployed  United 
Kingdom  computer  system  administrator"  broke  into  a  U.S.  Naval 
Weapons  Station  computer  network,  stealing  computer  passwords,  and 
shutting  down  the  network  [1].  This  was  not  an  isolated  event  [2],  and  as 
the  Navy  moves  aggressively  toward  network- centric  warfare,  concerns 
with  protecting  its  networks  have  become  critical. 

In  response  to  the  increasing  threats  and  risk  associated  with  Department 
of  Defense  (DoD)  computer  networks,  the  Joint  Chiefs  of  Staff  issued  a 
memo  (CJCS  Memo  CM-5 10-99  dated  10  Mar  99)  that  offered  specific 
guidance  for  protecting  networks  against  threats  or  actual  attacks.  That 
memo  identified  five  levels  of  information  operations  conditions 
(INFOCONs)  that  progressively  protected  Navy  computer  networks. 
Research  to  assess  their  efficacy  [3]  was  conducted  and  determined  that 
INFOCONs  did  protect  networks,  but  with  proportional  loss  of  net¬ 
work  capability.  A  more  "surgical"  method  of  network  defense  was 
required,  and  the  current  research  was  designed  to  address  that  need 
using  the  Embedded  Firewall  (EFW)  concept  [4,  5]. 

EXPERIMENT  DESIGN 

SSC  San  Diego’s  Information  Operations  Center  of  the  Future  (IOCOF) 
conducted  the  Information  Operations  Command  and  Control  Dynamic 
Network  Defense  (IOC2  DND)  Experiment  over  3  days  in  November 
2002. 1  The  objective  of  the  experiment  was  to  assess  dynamic  network 
defense,  using  EFW,  while  measuring  the  impact  on  a  warfighter’s  work¬ 
load,  situational  awareness,  and  command  and  control  capabilities.  EFW 
is  a  Defense  Advanced  Research  Projects  Agency  (DARPA)-developed 
solution  commercialized  by  3Com©  that  adds  a  layer  of  protection 
across  the  network  via  a  policy2  server  and  network  interface  cards.  In 
contrast  to  the  traditional  INFOCON  approach  to  network  security, 
the  EFW  option  allows  centralized  firewall  management  and  dynamic 
response,  tailoring  defense  by  individual  server,  workstation,  or  laptop. 


ABSTRACT 

The  Information  Operations 
Command  and  Control 
Dynamic  Network  Defense 
(IOC2  DND)  experiment  was 
designed  to  assess  the  ability  to 
defend  a  computer  network 
dynamically  using  the 
Embedded  Firewall  (EFW) 
network  interface  system,  and 
to  measure  the  impact  on  a 
warfighter’s  workload,  situa¬ 
tional  awareness,  and  com¬ 
mand  and  control  capabilities. 
SSC  San  Diego’s  Information 
Operations  Center  of  the 
Future  provided  a  highly 
authentic  shipboard  joint 
operations  center  environment 
and  task  scenario,  defending 
against  simulated  hacker 
attacks  using  a  realistic  net¬ 
work  and  EFW  rule  set  experi¬ 
mentation.  This  paper  discusses 
the  IOC2  DND  experiment 
and  its  results. 


1  Experiment  sponsors  included  OPNAV  N64,  Navy  Warfare  Development 
Command  (NWDC),  Defense  Advanced  Research  Projects  Agency  (DARPA), 
Space  and  Naval  Warfare  Systems  Command  (SPAWAR)  PMW-161,  and 
SPAWAR  PMW-189. 

2  The  word  "policy"  in  this  context  refers  to  an  EFW  rule  set  and  has  no  impli¬ 
cations  for  organizational  policy. 
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The  IOCOF  provided  a  highly  authentic  replication  of  a  shipboard  joint 
operations  center  (JOC),  patterned  after  the  JOC  aboard  USS  Coronado 
(AGF  11),  the  Third  Fleet’s  Flagship.  The  JOC  included  such  systems  as 
Global  Command  and  Control  System-Maritime  (GCCS-M)  and 
Information  Technology  for  the  21st  Century  (IT-21)  workstations. 
Additionally,  the  IOCOF  incorporated  a  broader  test  bed  to  conduct 
detailed  technical  analysis. 

During  Phase  I,  the  SSC  San  Diego  Red  (network  attack)  and  Blue  (net¬ 
work  defense)  teams  worked  collectively  to  identify  the  ideal  combina¬ 
tion  of  EFW  rule  sets  and  policies  to  enable  a  Joint  Task  Force  (JTF)  to 
operate  during  network  attacks  on  a  simulated  afloat  mission. 

During  Phase  II,  volunteer  Commander  Third  Fleet  (C3F)  participants 
manned  the  IOCOF  as  a  JOC  for  five  4-hour  watches  over  3  days.  The 
watches  provided  a  backdrop  to  assess  the  impact  of  EFW  policies  in  an 
operational  environment.  Working  against  a  realistic  naval  scenario  and 
interacting  with  the  IOCOF  White  Cell,  JOC  watchstanders  performed 
roles  typical  of  those  experienced  during  JTF  operations  afloat.  The  focus 
of  this  effort  was  to  develop  tactics,  techniques,  and  procedures  to  aid 
EFW  transition  into  the  Fleet.  Experiment  personnel  accumulated  over  a 
gigabyte  of  electronic  data  during  each  watch  and  gathered  metrics  from 
participant  observations  and  surveys. 

WATCHES 

Although  the  scenario  theme  was  consistent  across  the  five  watches,  each 
watch  was  distinct  in  that  the  specific  tasks  and  resources  differed.  In 
addition,  the  five  watches  incorporated  five  different  EFW  conditions,  as 
follows: 

•  Watch  1  -  Baseline  EFW  policy.  During  Phase  I,  this  rule  set  was 
determined  to  provide  maximal  network  protection  while  assuring  no 
expected  impact  to  warfighters. 

•  Watch  2  -  Baseline  EFW  policy  plus  client  file  sharing  blocked  (same 
policy  in  effect  for  entire  watch).  This  is  considered  an  optimal 
"hardened"  policy,  again  having  minimal  impact. 

•  Watch  3  -  A  denial  of  service  (DoS)  exploit  was  initiated  and  then 
compounded  by  denying  access  to  the  backup  domain  controller 
(simulating  pushing  the  wrong  anti-DoS  rule  set).  Later,  the  correct 
rule  set  was  "pushed,"  with  full  service  restored  to  all  systems  in  the 
second  half. 

•  Watch  4  -  Started  with  the  baseline  hardened  EFW  policy,  and  then 
approximately  half-way  into  the  watch  the  selective  minimize  policy 
was  initiated. 

•  Watch  5  -  Started  with  the  baseline  hardened  EFW  policy,  and  then 
approximately  30  minutes  into  the  watch  all  chat  was  blocked. 

RESEARCH  PARTICIPANTS 

The  experiment’s  nine  participants  consisted  of  six  Navy  officers  and 
three  chief  petty  officers  with  166  years  of  combined  military  service.  The 
average  age  was  38.8  years,  and  their  mean  years  of  service  was  18.4.  The 
participants  served  in  the  simulated  JOC  as  a  Combined  Joint  Task  Force 
(CJTF)  team  during  the  3  days  of  simulated  operations.  Research  data 
were  collected  and  analyzed  from  the  participants. 
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METRICS 

Workload  was  measured  using  the  National  Aeronautics  and  Space 
Administration  (NASA)  Task  Load  Index,  a  standardized  form  address¬ 
ing  level-of-effort  topics  [6].  On  the  same  form,  research  participants 
were  asked  to  provide  their  estimates  of  (1)  situational  awareness  (SA), 

(2)  task-related  command  and  control  capability,  and  (3)  supporting 
team’s  computer  network  availability,  all  using  a  10-point  scale  that 
ranged  from  1  (poor)  to  10  (outstanding).  In  addition,  because  it  is  critical 
to  practically  all  military  experiments  and  exercises,  SA  [7]  was  assessed 
during  each  watch  using  the  Situation  Awareness  Rating  Technique 
(SART).  SART  assesses  each  respondent’s  task-related  demand  (cognitive 
demand),  supply  (resources),  and  understanding,  which  combined  mathe¬ 
matically  yields  an  index  of  SA  such  that  when  demand  exceeds  supply, 
there  is  a  negative  effect  on  understanding,  leading  to  an  overall  reduction 
in  SA.  Thus,  two  different  estimates  of  SA  were  obtained  during  each 
watch,  namely  a  subjective  measure  (1  to  10)  on  the  NASA  Task  Load 
Index  form,  and  the  SART  One  other  metric  was  obtained  from  each 
participant:  the  amount  of  time  it  would  take  to  complete  his  or  her  task 
at  the  end  of  that  watch.  The  reasoning  was  that  under  typical  conditions, 
when  SA  was  high,  and  networks  were  normal,  JOC  participants  could 
generally  maintain  task  requirements  and  remain  "  ahead  of  the  curve. " 
However,  when  SA  was  reduced  and/or  networks  were  degraded,  the 
workload  demand  would  require  increased  time  to  complete  assigned 
tasking. 

DATA  ANALYSIS 

The  data  were  collected  during  the  five  watches;  all  findings  reported  here 
are  stated  as  descriptive  statistics,  without  inferential  statistical  implica¬ 
tions.  To  obtain  a  better  comparison  among  these  various  data,  each  hav¬ 
ing  different  means  and  variance  metrics,  the  metrics  were  converted  to 
McCall’s  T-scores  [8]  to  facilitate  their  comparison  across  measures  and 
within  watches  (W);  the  data  are  shown  in  Table  1 .  The  T-score  is  stan¬ 
dardized,  having  a  mean  of  50  and  a  standard  deviation  of  10,  thus  allow¬ 
ing  comparison  across  different  impact  indices. 

These  data  clearly  show  that  during  Watches  1  and  2,  when  the  baseline 
and  enhanced  baseline  EFW  policies  were  in  effect,  participants  reported 
few  or  no  adverse  effects  in  either  workload  or  for  the  various  measures 
of  SA.  That  finding  holds  also  for  the  participants’  estimates  of  command 


TABLE  1 .  Standardized  (T-scores)  operational  capability  measures. 


1 

Day  1 

Day  2 

Day  3 

1 

Mean 

W1 

W2 

W3  Early 

W3  Late 

W4 

W5 

Dependent 

Variables 

Baseline 

EFW 

Baseline 
EFW  + 

DOS 

Baseline 

EFW 

Minimized 

Chat 

Blocked 

Workload 

46.9 

46.4 

70.3 

44.5 

44.8 

47.1 

50.0 

SA:  (1-10) 

48.9 

54.5 

30.5 

57.2 

56.1 

52.8 

50.0 

SA:  SART 

51.2 

57.2 

30.7 

56.7 

55.3 

48.8 

50.0 

C2  Capability 

51.1 

52.5 

30.2 

57.6 

55.7 

52.9 

50.0 

Increased  Time 

1 

44.4 

43.1 

68.8 

42.5 

48.3 

53.0 

50.0 

1 
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and  control  (C2)  capability.  Moreover,  SA  increased  slightly  in  Watch  2, 
consistent  with  the  powerful  group  "learning  effect"  noted  in  the  first 
IOC2  experiment  [3]. 

Perhaps  the  most  striking  finding  during  this  phase  of  the  experiment 
occurred  early  on  the  second  day,  during  Watch  3,  when  a  simulated  DoS 
attack  occurred  and  the  EFW  DoS  policy  was  initiated  early  in  the  watch 
and  enforced  for  approximately  70  minutes.  After  that,  the  DoS  policy 
was  removed,  and  the  baseline  EFW  policy  was  instituted.  The  data  in 
Table  1  show  the  impact  to  the  participants  of  the  EFW  policy  in  effect 
during  the  first  half  of  Watch  3.  Workload  increased,  and  all  measures  of 
SA  declined  precipitously,  as  did  the  participants’  estimates  of  their  C2 
capability. 

Similarly,  the  participants’  estimated  increased  time  to  complete  their 
tasks  rose  to  the  highest  level  in  the  experiment,  approximately  a 
nineteen-fold  increase  over  baseline  measures.3 
Although  not  shown,  and  perhaps  consistent  with 
expectations,  it  is  interesting  to  note  that  the  various 
measures  of  phone  use  increased  during  this  time, 
approximately  three-fold  for  all  measures. 

These  data  clearly  show  the  two  standard  deviation 
declines  in  all  measures  of  SA  and  C2  capability,  as 
well  as  the  equally  substantial  increases  to  workload 
and  estimates  of  increased  time  during  the  early  part 
of  Watch  3.  Any  data  shifts  that  approximate  two 
standard  deviations  must  be  considered  important. 

Note  also  that  all  measures  in  Table  1  returned  to 
their  approximate  baseline  levels  during  the  second 
half  of  Watch  3.  The  SA  and  workload  T-score 
data  are  shown  graphically  for  all  five  watches  in 
Figure  1. 

In  general,  modifications  to  the  network  using  EFW 
in  response  to  mission  requirements  were  accom¬ 
plished  easily  and  quickly  from  the  EFW  policy 
server.  Additionally,  laboratory  EFW  implementa¬ 
tion  on  operational  networks  resulted  in  numerous 
lessons  learned  to  assist  in  fleet  transition  and 
employment. 


CONCLUSIONS 

To  protect  networks,  some  services  and  ports  must 
be  restricted,  yet  the  extent  of  restriction  required 

empirical  research  and  analysis.  Based  on  this  research  we  conclude  that: 

•  The  data  in  Table  1  and  Figure  1  indicate  that  the  EFW  policies 
chosen  for  this  experiment,  with  one  exception,  likely  will  not  pose 
negative  impact  for  operational  warfighters. 

•  EFW  rule  sets  were  effective,  with  little  or  no  disruption  to  the 
operators. 

•  The  EFW  proved  beneficial  in  managing  and  reallocating  bandwidth 
and  enforcing  operations  security. 

•  Dynamic  network  defense,  using  EFW,  allowed  immediate,  surgical, 
concrete  responses  to  network  threats. 


FIGURE  ] .  Standardized  impact  measures:  situation  awareness  and 
workload. 
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Finally,  one  participant’s  post-experiment  comment  indicates  the  overall 
value  of  the  IOC2  DND  experiment: 

"This  experiment  was  critical  to  C3F  watchstanders  and  afforded  us  the 
ability  to  run  a  full  CJTF  scenario  exercise  while  our  computer  networks 
were  undergoing  attacks.  This  controlled  environment  afforded  us  the 
ability  to  experience  what  it  would  be  like  to  conduct  operations  in  a 
hostile  network  environment.  We  would  not  have  been  able  to  do  this  at 
sea." 
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INTRODUCTION 

The  process  for  conducting  operational  risk  assessments  described  here 
was  developed  in  response  to  a  bona  fide  requirement  to  mitigate  the  risk 
associated  with  the  acquisition  and  fielding  of  C4ISR  technologies  that 
undergo  rapid  rates  of  advancement.  The  process1  provides  critical  opera¬ 
tional  information  for  technology  developers  and  assists  in  the  speed  and 
efficiency  at  which  a  technology  is  fielded  by  leveraging  Fleet  and  Marine 
warfighter  input  and  collaborating  with  the  technology  developer  to  iden¬ 
tify  critical  operational  issues  and  define  measurement  standards  (i.e., 
measures  of  effectiveness,  suitability,  and  performance). 

The  process  was  developed  with  the  support  of  Operational  Test  and 
Evaluation  Force  (OPTEVFOR)  and  is  designed  to  complement  their 
operational  assessment  and  evaluation  processes.  Test  plans  and  reports 
produced  during  the  process  are  designed  to  match  those  of  OPTEVFOR 
to  maintain  continuity  if  and  when  a  technology  transitions  to  a  program 
of  record.  The  documentation  of  critical  operational  issues,  measures  of 
effectiveness,  measures  of  suitability,  test  plans,  and  reports  can  assist  an 
OPTEVFOR-conducted  operational  evaluation  and,  ultimately,  the  speed 
at  which  even  large,  program  of  record  technologies  are  fielded. 

A  fundamental  aspect  of  the  process  is  the  early  and  continuous  involve¬ 
ment  of  the  warfighter  with  the  technology  developer,  sponsor,  and  other 
stakeholders.  Through  this  teaming,  technologies  can  effectively  be  fielded 
with  as  much  risk  mitigated  as  possible,  and  can  positively  influence 
speed  to  capability.  This  process  was  primarily  developed  to  be  used 
during  operational  testing  and  acquisition,  but  the  fundamentals  can  be 
applied  to  assessments  conducted  during  developmental  testing.  The 
process  has  two  sub-processes,  very  early  and  early  operational  risk 
assessments  (VEORA  and  EORA)  that  can  be  conducted  during  the 
concept  development  phase  of  both  evolutionary  acquisition  models. 
Figure  1  shows  the  evolutionary  acquisition  process  as  described  by  the 
traditional  incremental  developmental  model  and  the  spiral  development 
model  as  derived  from  DoD  Directive  5000.1  [1]  and  DoD  Instruction 
5000.2  [2]  of  12  May  2003.  Throughout  spiral  development,  OPTEVFOR 
will  operate  more  dynamically  ad  assess  the  risks  associated  with  each 
increment.  [3] 


ABSTRACT 

This  paper  describes  a  process 
that  mitigates  the  risk  associ¬ 
ated  with  the  acquisition  and 
fielding  of  command,  control, 
communications,  computers, 
intelligence,  surveillance  and 
reconnaissance  (C4ISR) 
technologies  to  the  Fleet.  By 
leveraging  warfighter  input 
early  and  working  with  the 
technology  developer  to 
identify  critical  operational 
issues  and  define  measurement 
standards,  the  process  assists 
in  the  speed  and  efficiency 
at  which  a  technology  is 
fielded. 


1  For  simplicity,  the  process  for  conducting  operational  risk  assessments  on 
command,  control,  communications,  computers,  intelligence,  surveillance  and 
reconnaissance  (C4ISR)  technologies  will  be  referred  to  in  this  paper  as  "the 
process." 
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VEORA  -  Very  Early  Operational  Risk  Assessment 

EORA  -  Early  Operational  Risk  Assessment 

ORA  -  Operational  Risk  Assessment 

EOA  -  Early  Operational  Assessment 

OA  -  Operational  Assessment 

OPEVAL-  Operational  Evaluation 

FOTE  -  Follow-On  Test  and  Evaluation 

IOC  -  Initial  Operational  Capability 

FOC  -  Full  Operational  Capability 


LRIP-  Low-  Rate  Initial  Production 

IOT&E  —  Initial  Operational  Test  and  Evaluation 

ERP  -  Full-Rate  Production 

NSS  -  National  Security  Strategy 

NMS  -  National  Military  Strategy 

ICD  -  Initial  Capabilities  Document 

JROC  -  Joint  Requirements  Oversight  Committee 

CD  -  Concept  Development 

AoA  -  Analysis  of  Alternatives 


MS  -  Milestone 

CDD  -  Capability  Development  Document 
CPD  -  Capability  Production  Document 
DAB  -  Defense  Acquisition  Board 
DOTMLPF  -  Doctrine,  Organization,  Training,  Materiel, 
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Personnel,  Facilities 


FIGURE  1.  The  evolutionary  acquisition  process. 


OPTEVFOR 

Statute  dictates  that  in  each  of  the  services  only  one  independent  com¬ 
mand  will  have  the  charter  to  conduct  assessments  prior  to  fielding  an 
acquisition;  thus,  the  process  was  developed  with  the  aid  and  support  of 
OPTEVFOR.  Additionally,  OPTEVFOR  has  been  conducting  operational 
assessment  for  50  years;  thus,  it  made  sense  to  seek  guidance  and  work 
collaboratively  with  OPTEVFOR  in  the  development  of  the  process.  The 
process  is  intended  to  complement  OPTEVFOR’s  assessment  process, 
not  replace  or  compete  with  it.  This  is  partially  accomplished  by  creating 
test  plans  and  reports  that  have  the  same  look  and  feel  as  those  of 
OPTEVFOR’s.  The  metrics  section  explains  this  in  further  detail. 
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CRITICAL  OPERATIONAL  ISSUES 

Critical  operational  issues  are  key  to  ensuring  that  an  assessment  focuses 
on  what  is  important  to  the  warfighter.  Critical  operational  issues  are  pre¬ 
sented  in  the  form  of  a  question,  and  are  answered  by  process  metrics. 
The  process  identifies  critical  operational  issues  through  a  collaborative 
effort  in  one  or  more  planned  sessions.  The  technique  used  involves  a 
subject-matter  expert  briefing  a  group  of  selected  individuals  while  they 
input  their  responses  into  a  computer-based  collaborative  tool.  In  the 
near  future,  this  part  of  the  process  will  be  almost  entirely  web-based. 

The  tool  allows  for  simultaneous  but  anonymous  input  and  ensures  that 
all  data  and  participant  comments  are  collected  and  archived.  By  allowing 
simultaneous  input  during  a  brief,  a  subject-matter  expert  can  answer 
questions  and  clarify  functionality  as  to  a  technology’s  capabilities  and 
limitations.  Anonymity  allows  for  a  broad  spectrum  of  personnel  to 
work  collaboratively  without  rank,  position,  or  seniority  being  a  factor. 
For  example,  a  group  consisting  of  a  junior  officer,  a  senior  enlisted  per¬ 
son,  a  systems  engineer,  and  a  field  technician  can  provide  input  without 
the  influence  of  any  disparity  in  rank  or  pay  grades.  At  the  conclusion  of 
each  session,  the  group  votes  to  determine  the  most  important  issues.  The 
output  is  collected  and  forms  the  basis  of  the  metrics  to  be  used. 


METRICS 


Assessments,  of  any  type,  must  have  a  method  of  measuring  the  outcome 
of  a  test.  Test  objectives  are  obtained  by  the  collaborative  development  of 
measures  of  effectiveness,  measures  of  suitability,  and/or  measures  of  per¬ 
formance.  With  the  exception  of  measures  of  performance,  these  are 
defined  and  developed  in  the  same  way  as  OPTEVFOR. 

Measures  of  effectiveness  are  developed  based  on  how  well  a  technology 
is  designed  to  perform  in  its  intended  environment,  with  as  many  opera¬ 
tional  aspects  taken  into  consideration  as  possible.  For  example,  if  a  tech¬ 
nology’s  mission  is  to  provide  area  surveillance,  then  a  test  to  determine 
its  effectiveness  would  be  to  install  it  where  it  is  expected  to  be  fielded, 
with  consideration  as  to  what  might  diminish  its  capabilities,  such  as  the 
environment,  expected  threat,  and  countermeasures. 

Measures  of  suitability  are  defined  as  the  capability  of  a  technology,  when 
operated  by  warfighters,  to  perform  as  expected.  Measures  of  suitability 
have  been  categorized  by  OPTEVFOR  and  are  referred  to  as  suitability 
tests.  Suitability  test  categories  are  intuitive  and  guide  the  test  developers 
in  placing  the  warfighter’s  issues  in  the  categories  listed  below: 


S-l  Reliability 
S-3  Availability 
S-5  Compatibility 
S-7  Training 
S-9  Safety 


S-2  Maintainability 

S-4  Logistical  Supportability 

S-6  Interoperability 

S-8  Fluman  Factors 

S-l  0  Documentation 


At  the  completion  of  the  collection  of  critical  operational  issues,  it  is 
determined  whether  a  measure  of  effectiveness  or  a  measure  of  suitability 
will  be  best  suited  to  satisfy  a  particular  critical  operational  issue. 


Normally,  measures  of  performance  are  not  used  by  OPTEVFOR  in 
their  assessment  processes.  However,  as  defined  in  the  context  of  this 
process,  measures  of  performance  are  a  viable  metric.  Additionally,  other 
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organizations  in  the  test  and  evaluation  community 
define  and  use  measures  of  performance.  The 
process  defines  measures  of  performance  as  to  how 
well  a  technology  functions  in  relation  to  a  previous 
version  or  release.  They  are  used  to  effectively  meas¬ 
ure  the  differences  in  engineering  and  technological 
changes  from  the  warfighter’s  perspective.  This  can 
be  a  positive  or  negative  change  in  performance,  but 
as  with  any  other  measure,  must  not  be  weighed 
alone. 

For  a  more  detailed  explanation  of  OPTEVFOR 
processes,  procedures,  and  terminology,  refer  to  [1]. 

TEST  EXECUTION 

A  test  plan  is  produced  with  metrics  designed  to  sat¬ 
isfy  the  identified  critical  operational  issues,  i.e.,  the 
tests  answer  the  questions  raised  by  the  operational 
issues.  The  test  plan  will  also  include  the  schedule  of 
events,  identification  of  the  assessors,  any  security  issues,  and  a  scenario 
where  necessary.  A  dry  run  is  conducted  for  a  final  shakeout  and,  within 
the  constraints  of  the  schedule,  the  subject-matter  expert  will  determine 
that  the  technology  is  ready  to  test.  The  technology  is  then  given  to  the 
warfighter  (Figure  2).  Testing  is  best  conducted  in  two  3-hour  sessions 
each  day  of  the  assessment.  An  assessment  should  take  place  over  a  2-  to 
5-day  period,  depending  on  the  complexity  of  the  technology  and  the 
number  of  identified  critical  operational  issues.  Back-up  sessions  can  be 
scheduled  due  to  operational  issues,  and  follow-on  sessions  can  be  sched¬ 
uled  if  other  serious  issues  are  identified  during  the  primary  assessment. 

DATA  COLLECTION  AND  ANALYSIS 

Data  sets  are  collected  using  a  government-owned,  web-based  tool,  the 
Joint  Battle  Center’s  Data  Collection  and  Analysis  Tool  (JDCAT). 
JDCAT  is  used  for  data  collection,  analysis,  and  as  an  aid  in  the  develop¬ 
ment  of  the  final  report.  Analysis  is  performed  by  mapping  critical  opera¬ 
tional  issues  to  metrics,  then  directly  to  the  warfighter  evaluation.  The 
captured  data  sets  are  irrefutable.  Each  response  from  the  warfighter  cor¬ 
relates  to  a  collaboratively  developed  and  agreed-upon  critical  operational 
issue  that  maps  directly  to  the  chosen  measurement  of  effectiveness,  suit¬ 
ability,  or  performance. 

FINAL  REPORT 

The  final  report  is  a  culmination  of  the  identified  critical  operational 
issues,  metrics,  test  plan,  layout,  outcome,  procedures  used,  and  summary. 
The  final  report  may  include  a  recommendation  by  the  facilitators  of  the 
assessment  (this  will  be  determined  before  the  process  starts).  The  final 
report  is  designed  to  look  and  feel  similar  to  reports  produced  by 
OPTEVFOR.  Thus,  if  and  when  a  technology  becomes  a  program  of 
record  and  comes  under  the  auspices  of  OPTEVOR,  the  report  will  aid 
the  operational  test  director  in  determining  on  what  areas  and  functional¬ 
ities  an  operational  evaluation  must  focus. 
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CONCLUSION 

Although  this  process  was  developed  with  Navy  and  Marine  warfighters 
in  mind,  its  fundamentals  and  concepts  can  apply  to  all  branches  of  service 
as  well  as  our  North  Atlantic  Treaty  Organization  (NATO)  and  coalition 
partners.  In  our  current  joint  and  coalition  environment,  our  goal  is  to 
involve  as  many  organizations  as  possible  for  not  only  adoption  of  this 
process,  but  for  participation  in  collaborative  sessions.  This  is  becoming 
more  of  a  possibility  because  of  web-enabled  tools,  multi-national  net¬ 
works,  and  language  translators. 
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INTRODUCTION 

Given  the  increasing  complexity  of  real-world  applications,  the  need  to 
access  a  collection  of  cooperating  but  autonomous  distributed  databases 
becomes  inevitable.  In  such  a  federated  database  system,  which  consists 
of  a  collection  of  databases  and  related  access  methods,  semantically  relat¬ 
ed  data  might  be  represented  in  different  database  schemas  under  diverse 
database  management  systems  (DBMSs).  The  quantity  of  data  is  increas¬ 
ing  with  no  end  in  sight,  and  it  has  been  estimated  that  the  amount  of 
data  stored  in  the  world’s  databases  doubles  every  20  months  [1]. 
Inexpensive,  multi-gigabyte  disks  and  other  storage  devices  allow  us  to 
save  much  data.  However,  our  ability  to  interpret  and  analyze  the  data  is 
still  limited.  Thus,  data  mining  is  relegated  as  one  of  the  few  paths  for 
elucidating  patterns  from  the  data  [2].  Nevertheless,  the  great  quantity  of 
data  makes  the  discovery  process  computationally  expensive.  It  is  desir¬ 
able  to  effect  the  knowledge  discovery  process  on  a  relatively  con¬ 
strained  subset  of  data  to  reduce  the  computational  complexity.  Here, 
domain  knowledge  can  be  used  to  reduce  the  rank  of  the  data  being  con¬ 
sidered  to  limit  the  search  for  patterns  [3].  It  is  well  known  that  the  user 
in  the  loop  has  some  previous  concepts  or  knowledge  about  the  domain 
represented  by  the  database. 

Domain  knowledge  is  predicated  upon  domain  representation.  A  proper 
representation  enables  the  extraction  of  features  for  use  in  data  mining.  It 
is  necessary  that  the  feature  space  be  randomized  [4]  — for  example,  in 
the  form  of  an  algorithm — to  offset  representational  complexity. 
However,  in  many  domains  of  military  interest— including  battle  man¬ 
agement  and  command,  control,  communications,  computers,  intelli¬ 
gence,  surveillance,  and  reconnaissance  (C4ISR)— full  algorithmic  defini¬ 
tions  are  not  known.  The  discovery  of  latent  semantic  knowledge  in  het¬ 
erogeneous  federated  databases  provides  a  viable  alternative. 

In  this  study,  a  framework  is  presented  consisting  of  database  cluster 
hierarchies  [5]  and  semantic  relationship  discovery  [3  and  6]  by  using 
object-oriented,  associative  mining,  and  logical  (fuzzy)  reasoning  tech¬ 
niques  to  effectively  fuse  a  large-scale  federated  database  system.  Users 
can  incrementally  and  dynamically  access  the  pieces  of  information  they 


ABSTRACT 

The  advent  of  the  Homeland 
Defense  initiative  has  brought 
with  it  a  renewed  sense  of 
importance  of  database  man¬ 
agement  technologies.  Data¬ 
base  cluster  hierarchies,  based 
on  the  affinity  relationships  of 
databases,  allow  users  to  incre¬ 
mentally  and  dynamically 
access  needed  information.  In 
this  study,  a  new  data-mining 
framework  is  presented.  The 
framework  employs  database 
clusters  to  discover  novel 
semantic  relationships  in  the 
object  classes  in  distinct  data¬ 
bases.  The  approach  uses 
object-oriented,  association 
rule  mining,  and  logical 
(fuzzy)  reasoning  techniques  to 
bridge  heterogeneity  in  large- 
scale  heterogeneous 
database  environments. 
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want  without  being  overwhelmed  with  all  of  the  unstructured  informa¬ 
tion.  The  proposed  framework  provides  a  flexible  means  for  sharing 
information  across  multiple  databases. 

DATABASE  CLUSTER  HIERARCHY 

Let,  Q={qi,  q2,---,qq]  contain  all  the  queries  issued  to  D={dj,  d2,...,dj] 
databases  in  a  period  of  time.  Let  M  be  a  symmetric  matrix  of  size  gxg 
indicating  the  affinity  measures  of  the  databases  in  a  partition  that  con¬ 
tains  g  databases  ( g  <  |Z)|).  Our  proposed  method  [5]  takes  the  training 
data  as  input,  computes  the  entities  of  the  matrix  M,  calculates  the  dis¬ 
tance  difference  values,  permutes  its  columns,  and  then  generates  an 
updated  matrix  M’.  The  permutation  is  performed  by  considering  the 
minimum  of  the  distance  difference  values  for  columns  i  and  j.  Let  col¬ 
umn  i  be  the  one  that  needs  to  be  placed  in  the  temporary  matrix  O  that 
consists  of  the  first  several  columns  of  M.  Column  i  can  be  placed  on  the 
left  or  right  of  column  j  in  O.  The  main  idea  is  that,  given  relative  stability 
in  the  domain  set,  Q,  position  column  i  in  the  place  that  satisfies  two 
conditions:  its  affinity  measure  should  be  less  than  or  equal  to  the  affinity 
measure  of  its  left  neighbor  and  greater  than  or  equal  to  the  affinity 
measure  of  its  right  neighbor.  Simply  consider  one  of  these  two  condi¬ 
tions  for  the  leftmost  or  the  rightmost  position  of  O  because  it  has  only 
one  neighbor  here.  The  mean  value  of  the  first  column  is  chosen  to  be  the 
splitting  criterion,  since  the  first  column  tends  to  have  the  larger  affinity 
value.  Two  clusters  can  thus  be  iteratively  generated  until  some  pre¬ 
defined  conditions  are  met. 

SEMANTIC  RELATIONSHIP  DISCOVERY 

A  data-mining  approach  consisting  of  association  rule  mining  and  logical 
reasoning  is  proposed  to  exploit  new  semantic  relationships  among  object 
classes  in  the  databases  for  each  cluster.  Clearly,  discovering  new  semantic 
relationships  for  object  classes  across  multiple  databases  will  not  only 
serve  to  facilitate  schema  integration  but  will  also  speed  up  query  pro¬ 
cessing. 

Generalized  Association-Rule-Mining  Method 

Figure  1  depicts  the  architecture  of  the  generalized 
association-mining  algorithm  where  domain 
knowledge  has  been  incorporated  [3].  The  idea  is 
to  use  domain  knowledge  to  eliminate  unneces¬ 
sary  computational  efforts  in  Phase  I  by  reducing 
the  rank  of  the  data  in  the  computation  tasks. 

To  evaluate  the  performance  of  the  experiments, 
five  real  databases  were  employed,  representing  22 
object  classes,  accessed  by  17,222  queries.  The 
total  number  of  operative  reductions  exceeded 
20,005,000,  which  represented  a  significant  sav¬ 
ings.  Several  experiments  were  conducted  by  vary¬ 
ing  the  number  of  databases,  object  classes,  or 
queries  to  make  the  performance  analysis  more 
general.  The  number  of  operations  in  one  particu¬ 
lar  step  in  the  first  phase  was  compared  with  and 
without  the  domain  knowledge,  as  shown  in 


FIGURE  1 .  Architecture  for  generalized  association  rule  mining  with 
domain  knowledge. 
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Figure  2(A,  B,  and  C).  The  fewer  the  operations,  the  better  would  be  the 
performance. 

ft  can  be  gleaned  from  Figure  2  that  the  number  of  operations  was  signif¬ 
icantly  reduced  under  all  of  the  experiments.  In  Figure  2(A),  the  number 
of  operations  required,  without  domain  knowledge,  using  various  num¬ 
bers  of  databases,  did  not  change.  This  followed  because  the  calculation 
was  based  on  the  number  of  object  classes  and  queries,  which  were  fixed. 
Using  domain  knowledge,  the  variations  in  the  number  of  operative 
reductions  was  small  when  the  number  of  databases  was  greater  than  50. 
However,  when  the  number  of  databases  was  less  than  50,  the  fewer  the 
number  of  databases,  the  larger  the  number  of  operative  reductions.  This 
reduction  can  be  explained  by  the  ratio,  |Q|/|D|.  That  is,  more  queries 
were  needed  to  partially  order  the  larger  databases.  Figure  2(B)  shows 
that  the  larger  the  number  of  object  classes,  the  larger  the  difference 
between  the  two  algorithms.  In  addition,  the  reduction  in  operations 
grew  exponentially  (as  can  be  seen  from  the  figure).  In  Figure  2(C),  the 
number  of  operative  reductions  increased  as  a  function  of  the  number  of 
queries,  and  the  increase  was  approximately  linear. 

Logical-Based  Reasoning 

Here,  a  logical  reasoning-based  mechanism  is  presented  for  the  inference 
of  new  semantic  relationships  in  two  databases  [6]. 

Definition  1.  h(Cs£)  Cuv)  is  the  logical  reasoning  function  that  derives  the 
new  semantic  relationships  between  two  object  classes  Cst  and  Cuv  from 
different  databases,  where  s<u  and  v>\. 

h(Cj£>  CMV)  =  CR(Cst,  Cul)  0  CR(Cuj,  Cuv), 
where  0  is  the  logical  operator  A  and  is  applied  to  each  element  in  CR. 

Definition  2.  An  object  class  relationship  CR(Cst,  Cuv)  represents  the 
superclass,  subclass,  and  equivalence  semantic  relationships  of  two  object 
classes  Cst  and  Cuv.  Its  value  is  captured  through  a  triplet  {P,  B,  E)  where 
P,  B,  and  E  indicate  the  suPerclass,  suBclass,  and  Equivalence  relations 
between  Cst  and  Cuv,  respectively. 

Definition  3.  g (Cuv,  Cst)  is  the  object  class  relationship  inversion  function 
such  that 

CR{CUV,  Cst)  =  g (C„v,  Cst)  =  ( Bh  Ph  Ef)  if  CR{Cst,  Cuv)  =  (Pp  Bu  £;). 

Let  Seq  be  the  object  class  equivalence  relationship  set  (obtained  from  our 
generalized  association-mining  algorithm)  and  TRSpk  be  the  total  object 
class  relation  set  for  the  cluster  Pk.  The  proposed  relationship-derivation 
algorithm  will  incrementally  update  TRS pk  and  is  presented  next. 

For  any  two  object  classes  Cst  and  Cuv  in  the  databases  in  Pk,  where  s<u 
and  v>\  { 

if  (( Cst ,  Cuv)&  TRSpk)  { 

if  (3 Cxy  satisfying  (Cs£,  Cxy)G 

$eq) 

For  every  Cxy 

if  ((Cxy,  CHV)&  TRSpk)  CR(Cst,  Cuv)  =  CR(Cxy,  CMV)\ 
else  if  (x>u  |  y~>v )  CA(C££,  Cuf)  —  g(C^,  Cuf)j 
else  CR(Cst,  Cuv)  =  h(Cst,  CHV); 

TRSpk  =  TRSpk  U  (Cst,  Cuv); 


(B) 


(0 


FIGURE  2.  Comparison  of  the  numbers  of 
operations  with  domain  knowledge  (with 
DK)  and  without  domain  knowledge 
(without  DK). 
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This  algorithm  is  iteratively  executed  on  all  pairs  of  object  classes  in  a 
cluster.  The  semantic  relationships  discovered  in  each  cluster  can  be  used 
to  assist  in  the  integration  task  in  that  cluster.  In  addition,  this  algorithm 
should  be  applied  to  each  cluster  in  the  database  cluster  hierarchy. 

CONCLUSIONS 

This  study  presents  a  new  data-mining  framework  for  bridging  hetero¬ 
geneity  in  large-scaled,  heterogeneous,  federated  database  systems.  The 
proposed  framework  is  based  on  database  cluster  hierarchy  and  semantic 
relationship  discovery  in  multiple  databases.  The  database  cluster  hierar¬ 
chy  provides  different  levels  of  abstractions  for  users  to  incrementally 
and  dynamically  access  information.  The  number  of  platter  switches  for 
query  processing  can  be  reduced.  This  follows  because  a  set  of  databases 
belonging  to  a  certain  application  domain  is  placed  in  the  same  cluster 
and  needs  to  occur  in  sequence  on  some  query  access  path.  Moreover,  the 
constructed  clusters  can  be  used  as  the  unit  not  only  for  query  process¬ 
ing,  but  also  for  discovering  the  superclass,  subclass,  and  equivalence 
relationships,  which  are  achieved  by  the  associative-mining  and  logical- 
reasoning  methods.  During  this  knowledge-acquisition  process,  some 
semantic  conflicts  can  be  detected  and  subsequently  resolved  through  the 
schema  integration  process. 

FUTURE  WORK 

The  semantic  relationship  between  pairs  of  databases  is  evolved  through 
a  query-driven  mechanism.  In  particular,  the  symmetric  matrices  M2,  M2’, 
...,  Mr  ,  where  r  is  the  rank  of  the  matrix,  can  be  used  to  fuzzify  the 
semantic  relations  between  databases  m  and  n  in  terms  of  2,  3,  ...,  r-step 

reachability  [7].  Furthermore,  the  transitive  closure  of  M,  defined  by 

r 

Mr  =  U  Ml  defines  the  fuzzy  semantic  relation  between  m  and  n  for  a 

given  r,  where  r  is  taken  as  unity  for  purposes  of  this  paper.  The  transitive 
closure  of  M  allows  for  anticipatory  cluster  formation,  which  of  course 
speeds  access.  The  principle  of  temporal  locality  can  be  applied  to  remove 
database  pairings  that  were  not  temporally  co-accessed.  Note  that  if  this 
operation  is  not  performed,  then  the  system  will  rapidly  degenerate  to  a 
single  cluster,  since  almost  every  database  is  fuzzily  related  to  every 
other.  To  avert  this  problem,  define  a  unit  time  interval  for  time  t  such 
that  the  query  sets  QW  and  Q(f+1)  satisfy  the  relational  constraints, 

0<|QW  (T  Q(£+1)|<min(|Q(£)|,|Q(£+1)|).  The  left-hand  constraint  ensures 
continuity,  while  the  right-hand  constraint  ensures  proper  variation  over 
time.  Next,  define  the  complement  matrix,  M  ,  such  that  the  ( m ,  w)1*1 
entry  is  1,  just  in  case  databases  m  and  n  have  not  been  accessed  together 
during  the  last  time  interval,  and  otherwise  0.  Then,  the  iterative  transi¬ 
tive  closure  is  defined  by  M^t+ 1  >  =  -M^\  The  use  of  sparse  matrices 

and  distributed  (server)  processing  ensures  tractability  in  this  approach 
and  allows  for  associative  cluster  formation,  which  speeds  the  discovery 
of  semantic  relationships. 
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